[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492395862 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002351] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006737] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007047] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.010173] LSM: Security Framework initializing [ 0.012082] Yama: becoming mindful. [ 0.013063] SELinux: Initializing. [ 0.014094] *** VALIDATE selinux *** [ 0.023314] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028736] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029195] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030160] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] *** VALIDATE tmpfs *** [ 0.033540] *** VALIDATE proc *** [ 0.034348] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.036344] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038083] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040045] Spectre V2 : User space: Vulnerable [ 0.041015] Speculative Store Bypass: Vulnerable [ 0.044762] debug: unmapping init [mem 0xffffffffb6259000-0xffffffffb6260fff] [ 0.046215] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047888] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049035] ... version: 2 [ 0.050024] ... bit width: 48 [ 0.051021] ... generic registers: 4 [ 0.052020] ... value mask: 0000ffffffffffff [ 0.053025] ... max period: 00007fffffffffff [ 0.054027] ... fixed-purpose events: 3 [ 0.055020] ... event mask: 000000070000000f [ 0.057306] rcu: Hierarchical SRCU implementation. [ 0.059749] smp: Bringing up secondary CPUs ... [ 0.060702] x86: Booting SMP configuration: [ 0.061048] .... node #0, CPUs: #1 #2 #3 [ 0.064724] smp: Brought up 1 node, 4 CPUs [ 0.066031] smpboot: Max logical packages: 1 [ 0.067022] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.223811] node 0 deferred pages initialised in 151ms [ 0.227269] devtmpfs: initialized [ 0.229525] x86/mm: Memory block size: 128MB [ 0.233326] gcov: version magic: 0x41383552 [ 0.236277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.241114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243291] pinctrl core: initialized pinctrl subsystem [ 0.245205] [ 0.245844] ************************************************************* [ 0.248018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251016] ** ** [ 0.253014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.256017] ** ** [ 0.259015] ** This means that this kernel is built to expose internal ** [ 0.261013] ** IOMMU data structures, which may compromise security on ** [ 0.264017] ** your system. ** [ 0.267015] ** ** [ 0.270015] ** If you see this message and you are not debugging the ** [ 0.272014] ** kernel, report this immediately to your vendor! ** [ 0.275017] ** ** [ 0.278016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.281017] ************************************************************* [ 0.284734] NET: Registered protocol family 16 [ 0.286488] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.289070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.292075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.296098] cpuidle: using governor menu [ 0.297776] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.300620] PCI: Using configuration type 1 for base access [ 0.303126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.312147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.313051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.314221] cryptd: max_cpu_qlen set to 1000 [ 0.316000] ACPI: Added _OSI(Module Device) [ 0.318015] ACPI: Added _OSI(Processor Device) [ 0.319013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.321165] ACPI: Added _OSI(Processor Aggregator Device) [ 0.325675] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.332257] ACPI: Interpreter enabled [ 0.334073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.336016] ACPI: Using IOAPIC for interrupt routing [ 0.338141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.343392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.352000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.355042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.358017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.362116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.367806] acpiphp: Slot [2] registered [ 0.369214] acpiphp: Slot [5] registered [ 0.371102] acpiphp: Slot [6] registered [ 0.372127] acpiphp: Slot [7] registered [ 0.374124] acpiphp: Slot [8] registered [ 0.375099] acpiphp: Slot [9] registered [ 0.377138] acpiphp: Slot [10] registered [ 0.379173] acpiphp: Slot [3] registered [ 0.380113] acpiphp: Slot [4] registered [ 0.382137] acpiphp: Slot [11] registered [ 0.384118] acpiphp: Slot [12] registered [ 0.385163] acpiphp: Slot [13] registered [ 0.387114] acpiphp: Slot [14] registered [ 0.389145] acpiphp: Slot [15] registered [ 0.390112] acpiphp: Slot [16] registered [ 0.392106] acpiphp: Slot [17] registered [ 0.393120] acpiphp: Slot [18] registered [ 0.394120] acpiphp: Slot [19] registered [ 0.396100] acpiphp: Slot [20] registered [ 0.397131] acpiphp: Slot [21] registered [ 0.399113] acpiphp: Slot [22] registered [ 0.400097] acpiphp: Slot [23] registered [ 0.401070] acpiphp: Slot [24] registered [ 0.403153] acpiphp: Slot [25] registered [ 0.405117] acpiphp: Slot [26] registered [ 0.407122] acpiphp: Slot [27] registered [ 0.409106] acpiphp: Slot [28] registered [ 0.411102] acpiphp: Slot [29] registered [ 0.412079] acpiphp: Slot [30] registered [ 0.416171] acpiphp: Slot [31] registered [ 0.418068] PCI host bridge to bus 0000:00 [ 0.420020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.423027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.433032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.437038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.439209] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.443063] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.445377] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.455808] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.461948] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.464079] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.466020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.469022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.472449] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475873] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.477038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.479764] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.488024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.502035] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.508023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.515040] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.524017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.530021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.549015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.560480] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.576024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.583036] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.602029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.613925] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.621016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.627017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.642045] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.651348] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.662018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.668027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.688016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.699886] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.710021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.718020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.734022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.745358] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.766021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.779025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.800021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.812355] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.814418] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.817362] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.820524] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.823254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.829115] iommu: Default domain type: Passthrough [ 0.832073] SCSI subsystem initialized [ 0.833170] ACPI: bus type USB registered [ 0.835136] usbcore: registered new interface driver usbfs [ 0.837097] usbcore: registered new interface driver hub [ 0.839086] usbcore: registered new device driver usb [ 0.842198] pps_core: LinuxPPS API ver. 1 registered [ 0.844066] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.847064] PTP clock support registered [ 0.851046] EDAC MC: Ver: 3.0.0 [ 0.853136] PCI: Using ACPI for IRQ routing [ 0.854547] NetLabel: Initializing [ 0.856018] NetLabel: domain hash size = 128 [ 0.858013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.861084] NetLabel: unlabeled traffic allowed by default [ 0.863170] vgaarb: loaded [ 0.865280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.867020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.874305] clocksource: Switched to clocksource kvm-clock [ 0.982362] VFS: Disk quotas dquot_6.6.0 [ 0.984330] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.987420] *** VALIDATE ramfs *** [ 0.989055] *** VALIDATE hugetlbfs *** [ 0.991344] pnp: PnP ACPI init [ 0.993730] pnp: PnP ACPI: found 6 devices [ 1.015468] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.020267] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.022790] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.025566] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.028781] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.031900] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.035386] NET: Registered protocol family 2 [ 1.038720] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.044723] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.049385] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.055337] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.059234] TCP: Hash tables configured (established 65536 bind 65536) [ 1.062818] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.067193] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.070267] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.073921] NET: Registered protocol family 1 [ 1.077188] RPC: Registered named UNIX socket transport module. [ 1.079712] RPC: Registered udp transport module. [ 1.081531] RPC: Registered tcp transport module. [ 1.083433] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.086052] NET: Registered protocol family 44 [ 1.087976] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.090230] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.092624] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.095294] PCI: CLS 0 bytes, default 64 [ 1.097364] Unpacking initramfs... [ 2.593149] debug: unmapping init [mem 0xffffa0657cc54000-0xffffa0657ffbffff] [ 2.600175] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.602868] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.607568] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.131324] Initialise system trusted keyrings [ 3.133217] Key type blacklist registered [ 3.135459] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.145720] zbud: loaded [ 3.149232] *** VALIDATE nfs *** [ 3.150783] *** VALIDATE nfs4 *** [ 3.152836] pstore: using deflate compression [ 3.156767] Platform Keyring initialized [ 3.264770] NET: Registered protocol family 38 [ 3.266467] Key type asymmetric registered [ 3.268141] Asymmetric key parser 'x509' registered [ 3.270116] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.274114] io scheduler mq-deadline registered [ 3.276115] io scheduler kyber registered [ 3.278212] io scheduler bfq registered [ 3.280986] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.283801] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.286812] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.290135] ACPI: Power Button [PWRF] [ 3.297905] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.305360] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.320203] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.327373] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.349666] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.381579] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.412567] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.417807] Non-volatile memory driver v1.3 [ 3.419815] Linux agpgart interface v0.103 [ 3.453901] virtio_blk virtio1: [vda] 149488 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.457403] vda: detected capacity change from 0 to 76537856 [ 3.474436] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.478590] vdb: detected capacity change from 0 to 1073741824 [ 3.495266] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.498263] vdc: detected capacity change from 0 to 2621440000 [ 3.511851] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.515169] vdd: detected capacity change from 0 to 2621440000 [ 3.528693] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.531651] vde: detected capacity change from 0 to 4294967296 [ 3.545718] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.548030] vdf: detected capacity change from 0 to 4294967296 [ 3.554376] libphy: Fixed MDIO Bus: probed [ 3.560841] usbcore: registered new interface driver usbserial_generic [ 3.564133] usbserial: USB Serial support registered for generic [ 3.566985] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.571829] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.574125] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.577030] mousedev: PS/2 mouse device common for all mice [ 3.580079] rtc_cmos 00:05: RTC can wake from S4 [ 3.583247] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.587794] rtc_cmos 00:05: registered as rtc0 [ 3.590464] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.590979] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.596884] intel_pstate: CPU model not supported [ 3.597625] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.604089] hid: raw HID events driver (C) Jiri Kosina [ 3.606518] usbcore: registered new interface driver usbhid [ 3.608743] usbhid: USB HID core driver [ 3.610254] drop_monitor: Initializing network drop monitor service [ 3.613191] Initializing XFRM netlink socket [ 3.615822] NET: Registered protocol family 10 [ 3.620369] Segment Routing with IPv6 [ 3.622190] NET: Registered protocol family 17 [ 3.626063] mpls_gso: MPLS GSO support [ 3.632479] RAS: Correctable Errors collector initialized. [ 3.634354] AVX version of gcm_enc/dec engaged. [ 3.636017] AES CTR mode by8 optimization enabled [ 3.715845] sched_clock: Marking stable (3715812430, 0)->(4697158793, -981346363) [ 3.719539] registered taskstats version 1 [ 3.721804] Loading compiled-in X.509 certificates [ 3.724145] zswap: loaded using pool lzo/zbud [ 3.752020] Key type big_key registered [ 3.765480] Key type encrypted registered [ 3.767251] ima: No TPM chip found, activating TPM-bypass! [ 3.769513] ima: Allocated hash algorithm: sha1 [ 3.771385] ima: No architecture policies found [ 3.773179] evm: Initialising EVM extended attributes: [ 3.775182] evm: security.selinux [ 3.776425] evm: security.ima [ 3.777693] evm: security.capability [ 3.779175] evm: HMAC attrs: 0x1 [ 3.781761] rtc_cmos 00:05: setting system clock to 2026-08-22 06:30:44 UTC (1787380244) [ 3.788199] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.791897] debug: unmapping init [mem 0xffffffffb5f82000-0xffffffffb6258fff] [ 3.802114] Write protecting the kernel read-only data: 28672k [ 3.805842] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.808963] debug: unmapping init [mem 0xffffffffb4f14000-0xffffffffb4ffffff] [ 3.843862] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.853585] systemd[1]: Detected virtualization kvm. [ 3.855579] systemd[1]: Detected architecture x86-64. [ 3.857645] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.884303] systemd[1]: No hostname configured. [ 3.886616] systemd[1]: Set hostname to . [ 3.889110] random: systemd: uninitialized urandom read (16 bytes read) [ 3.892223] systemd[1]: Initializing machine ID from random generator. [ 4.032112] random: systemd: uninitialized urandom read (16 bytes read) [ 4.035450] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.040744] random: systemd: uninitialized urandom read (16 bytes read) [ 4.044303] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.053211] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.650660] device-mapper: uevent: version 1.0.3 [ 4.653685] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.327696] virtio_net virtio0 ens2: renamed from eth0 [ 5.361032] random: fast init done [ 5.405210] scsi host0: ata_piix [ 5.448255] scsi host1: ata_piix [ 5.450807] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.454268] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.971944] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.288036] random: crng init done [ 10.291767] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.640921] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.771814] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.032607] SELinux: Disabled at runtime. [ 12.092568] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.101300] systemd[1]: Detected virtualization kvm. [ 12.103664] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.614904] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.619118] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.624940] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.630289] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.634602] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.645532] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.651616] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 12.754626] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK [0[ 13.075890] squashfs: version 4.0 (2009/01/31) Phillip Lougher m] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Kernel Device Manager. [ 13.411680] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.416758] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.543805] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.556398] EDAC sbridge: Ver: 1.1.2 [ 15.096622] Key type dns_resolver registered [ 15.399054] NFS: Registering the id_resolver key type [ 15.401530] Key type id_resolver registered [ 15.403192] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg451-server login: [ 46.186367] libcfs: loading out-of-tree module taints kernel. [ 46.290982] Key type ._llcrypt registered [ 46.292376] Key type .llcrypt registered [ 46.399494] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 65.509240] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 67.460546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 67.512054] alg: No test for adler32 (adler32-zlib) [ 69.126639] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 70.338203] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 72.279169] Key type lgssc registered [ 73.936751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.843041] hrtimer: interrupt took 5029270 ns [ 91.682623] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.958135] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 145.700526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.761442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.106174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.147102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.261653] Lustre: lustre-MDT0000: new disk, initializing [ 147.335193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.356314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.022153] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.999885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.111128] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.145410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.149621] Lustre: Skipped 1 previous similar message [ 166.245214] Lustre: lustre-MDT0001: new disk, initializing [ 166.317133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.377771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.393080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 170.844244] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.320868] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.761598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.032313] Lustre: lustre-OST0000: new disk, initializing [ 185.042152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.048441] Lustre: 8439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.102165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.441305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 190.459790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 190.519949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.838281] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.700703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.799980] Lustre: lustre-OST0001: new disk, initializing [ 204.804756] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.818935] Lustre: 9513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.882907] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.585760] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.021929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.030540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.074617] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 222.854078] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.127973] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.844170] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.158407] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing yml_node [ 245.659840] Lustre: DEBUG MARKER: Client: 2.17.54.115 [ 248.396414] Lustre: DEBUG MARKER: MDS: 2.17.54.115 [ 250.970043] Lustre: DEBUG MARKER: OSS: 2.17.54.115 [ 252.483648] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 02:34:51 EDT 2026 [ 270.741743] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 271.882204] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 289.248828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 289.269229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.293423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.784656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.784684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.796220] Lustre: Skipped 1 previous similar message [ 290.807366] Lustre: Skipped 3 previous similar messages [ 292.828872] Lustre: server umount lustre-MDT0000 complete [ 295.772622] LustreError: 10385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380536 with bad export cookie 16715435885918350970 [ 295.775275] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.787381] LustreError: 10385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 295.904376] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.920062] Lustre: Skipped 1 previous similar message [ 295.927091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 295.937562] Lustre: Skipped 1 previous similar message [ 301.024113] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.026624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.046500] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 301.050159] Lustre: Skipped 1 previous similar message [ 306.154258] LustreError: 7975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.158418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.188375] LustreError: 7975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 306.206630] Lustre: Skipped 1 previous similar message [ 310.239135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 310.401249] Lustre: server umount lustre-MDT0001 complete [ 313.995809] Lustre: server umount lustre-OST0000 complete [ 317.685041] Lustre: server umount lustre-OST0001 complete [ 323.768446] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 332.345034] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 364.317762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.523622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.549459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.687385] Lustre: lustre-MDT0000: new disk, initializing [ 364.760370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.779629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.697107] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 376.673079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.793893] Lustre: 15063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 376.864205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 376.867559] Lustre: Skipped 1 previous similar message [ 376.938037] Lustre: lustre-MDT0001: new disk, initializing [ 377.013572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 377.053495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 377.062542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 380.264848] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 387.977897] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 392.527717] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 397.120524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 397.309497] Lustre: lustre-OST0000: new disk, initializing [ 397.313392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 397.320116] Lustre: 17031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 397.382329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.777075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 398.796482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 398.930657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 401.974802] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 409.527847] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 413.293083] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 413.452294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 416.943482] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 417.092819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 419.298483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 419.311860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.331747] Lustre: Skipped 1 previous similar message [ 419.341768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 419.348725] Lustre: Skipped 1 previous similar message [ 424.779574] Lustre: server umount lustre-OST0000 complete [ 428.514168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 428.524920] LustreError: Skipped 1 previous similar message [ 428.531922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.543762] Lustre: Skipped 1 previous similar message [ 433.875831] Lustre: server umount lustre-MDT0000 complete [ 434.657815] LustreError: 15075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 434.673632] LustreError: 15075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 437.020039] LustreError: 15858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380677 with bad export cookie 16715435885918352503 [ 437.027000] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 437.033699] LustreError: 15858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 437.329413] Lustre: server umount lustre-MDT0001 complete [ 442.799490] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 02:38:01 (1787380681) [ 458.294232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.618589] LustreError: 19904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.675372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.815230] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 463.841300] LustreError: 19905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.002397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.536627] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 480.884612] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 487.080379] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 489.578877] Lustre: Failing over lustre-MDT0000 [ 489.588796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.592072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.598777] Lustre: Skipped 2 previous similar messages [ 489.620336] Lustre: Skipped 6 previous similar messages [ 489.867527] Lustre: server umount lustre-MDT0000 complete [ 495.073177] LustreError: 19905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.090219] LustreError: 19905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 509.642662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.714337] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.022993] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 515.047455] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 515.052896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 515.068799] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 522.025477] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 530.400464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.404336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.433453] Lustre: Skipped 1 previous similar message [ 530.445020] Lustre: Skipped 1 previous similar message [ 540.127781] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 540.510512] Lustre: server umount lustre-MDT0000 complete [ 540.645641] LustreError: 19904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.658841] LustreError: 19904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 543.488462] LustreError: 19886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380784 with bad export cookie 16715435885918353917 [ 543.494939] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.505984] LustreError: 19886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 543.868965] Lustre: server umount lustre-MDT0001 complete [ 552.079607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.508320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.512756] Lustre: Skipped 2 previous similar messages [ 556.127947] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 564.980305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.440906] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 578.283100] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 583.484331] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 585.646643] Lustre: Failing over lustre-MDT0000 [ 585.710290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.724092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.934635] Lustre: server umount lustre-MDT0000 complete [ 587.745337] LustreError: 23239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.757605] LustreError: 23239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 604.128250] Lustre: 3636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787380828/real 1787380828] req@ffffa065cf224e00 x1874204097168000/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787380844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 604.152543] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.550553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.346761] LustreError: 3633:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa064c3247b80 x1874204097170816/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 613.623431] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 618.023584] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 618.991084] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 619.011268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 619.017536] Lustre: Skipped 1 previous similar message [ 619.037281] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 627.563090] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 634.336501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 634.349848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 634.363903] Lustre: Skipped 4 previous similar messages [ 646.111466] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 646.301259] Lustre: server umount lustre-MDT0000 complete [ 649.607852] LustreError: 23220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380890 with bad export cookie 16715435885918354855 [ 649.614845] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 649.625884] LustreError: 23220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 649.909646] Lustre: server umount lustre-MDT0001 complete [ 657.916097] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 02:41:36 (1787380896) [ 676.299295] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 688.069940] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 731.379246] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 741.719936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.009853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 742.089892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 742.237716] Lustre: lustre-MDT0000: new disk, initializing [ 742.410931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 742.422567] Lustre: Skipped 2 previous similar messages [ 742.464412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 746.877455] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 756.857320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.954577] Lustre: 29616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 757.313108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 761.817304] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 771.368339] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 776.940753] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 783.360614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 783.609724] Lustre: 31584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 784.805129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 784.845086] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 784.910778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 784.933498] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 789.716978] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 798.589889] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 804.139656] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 830.945237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 830.964687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.976220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 830.984918] Lustre: Skipped 4 previous similar messages [ 830.996074] Lustre: Skipped 6 previous similar messages [ 832.318792] Lustre: server umount lustre-OST0000 complete [ 850.400109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 850.567717] Lustre: server umount lustre-MDT0000 complete [ 854.546729] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381095 with bad export cookie 16715435885918355261 [ 854.548502] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 854.563620] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 855.136362] Lustre: server umount lustre-MDT0001 complete [ 866.417941] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 868.621832] Key type lgssc unregistered [ 868.902500] LNet: 33563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.909534] LNetError: 33563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.930729] LNet: Removed LNI 192.168.204.151@tcp [ 869.720158] Key type .llcrypt unregistered [ 869.724131] Key type ._llcrypt unregistered [ 894.658475] Key type ._llcrypt registered [ 894.662316] Key type .llcrypt registered [ 894.769351] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 895.574730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 895.670035] alg: No test for adler32 (adler32-zlib) [ 896.737303] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 896.954911] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 898.599215] Key type lgssc registered [ 899.791499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 908.071569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 908.090329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.682147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.079239] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 920.042241] LustreError: 34977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.070345] LustreError: 34977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 924.562373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.967178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 929.781505] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 938.404580] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 944.420820] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 951.611554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.965944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.089538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 958.584696] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 966.095566] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 969.402510] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 984.115542] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 02:47:02 (1787381222) [ 986.124461] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 988.354905] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 997.610617] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 02:47:15 (1787381235) [ 1006.069973] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 02:47:24 (1787381244) [ 1006.962521] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1015.078086] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1015.090700] Lustre: Skipped 2 previous similar messages [ 1017.041800] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 02:47:35 (1787381255) [ 1020.486464] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1020.494386] Lustre: Skipped 3 previous similar messages [ 1037.121562] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1037.127537] Lustre: Skipped 6 previous similar messages [ 1040.196505] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 02:47:58 (1787381278) [ 1043.347577] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1047.479487] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1058.418374] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 02:48:16 (1787381296) [ 1060.256508] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1060.263552] Lustre: Skipped 3 previous similar messages [ 1074.547608] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 02:48:32 (1787381312) [ 1092.582563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.587183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.595295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1092.598431] Lustre: Skipped 1 previous similar message [ 1094.901069] Lustre: server umount lustre-OST0000 complete [ 1099.234211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1099.241911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.263789] Lustre: Skipped 1 previous similar message [ 1099.279926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.818325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.830261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.394683] Lustre: server umount lustre-MDT0000 complete [ 1107.945505] LustreError: 35981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.965830] LustreError: 35981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1109.473174] LustreError: 39512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381350 with bad export cookie 15032445361245289141 [ 1109.490933] LustreError: 39512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1109.497794] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1110.048285] Lustre: server umount lustre-MDT0001 complete [ 1121.627294] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 1124.770170] Key type lgssc unregistered [ 1124.990630] LNet: 42841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.997150] LNetError: 42841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.007638] LNet: Removed LNI 192.168.204.151@tcp [ 1125.878271] Key type .llcrypt unregistered [ 1125.882857] Key type ._llcrypt unregistered [ 1142.420426] Key type ._llcrypt registered [ 1142.424126] Key type .llcrypt registered [ 1142.583217] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 1162.007369] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 1163.111388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1163.157358] alg: No test for adler32 (adler32-zlib) [ 1164.280875] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1164.548918] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 1166.215233] Key type lgssc registered [ 1167.451449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.707202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1208.755105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.068264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1210.097992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1210.182439] Lustre: lustre-MDT0000: new disk, initializing [ 1210.233343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.253877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1215.103111] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1226.068860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.258891] Lustre: 45997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1226.449361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1226.451499] Lustre: Skipped 1 previous similar message [ 1226.519862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1226.596830] Lustre: lustre-MDT0001: new disk, initializing [ 1226.741655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1226.777216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1226.791381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1231.606731] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1241.308873] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1246.525861] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1253.529628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1253.732567] Lustre: lustre-OST0000: new disk, initializing [ 1253.735982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1253.743613] Lustre: 47965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1253.846903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.064482] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1262.143406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1262.162371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1262.264309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1269.304673] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1273.959684] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1274.146879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1279.034224] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1279.214378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1283.050691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1283.055202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.063601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1283.070810] Lustre: Skipped 1 previous similar message [ 1283.090171] Lustre: Skipped 1 previous similar message [ 1287.576211] Lustre: server umount lustre-OST0000 complete [ 1293.282408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.282738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1293.304956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.306928] LustreError: Skipped 1 previous similar message [ 1293.319435] Lustre: Skipped 1 previous similar message [ 1298.401411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1298.411940] Lustre: Skipped 1 previous similar message [ 1302.497763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.506979] Lustre: Skipped 1 previous similar message [ 1306.079619] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1306.485991] Lustre: server umount lustre-MDT0000 complete [ 1309.701879] LustreError: 45990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381550 with bad export cookie 4926864653643031055 [ 1309.707962] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1309.712651] LustreError: 45990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1310.029267] Lustre: server umount lustre-MDT0001 complete [ 1318.666112] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 02:52:37 (1787381557) [ 1325.086434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.575485] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.607080] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1325.666362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.656203] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1330.656996] LustreError: 50427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.780715] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.042445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.517670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.857339] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1351.988308] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1357.914330] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1364.953891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1365.324281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.560849] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1378.897473] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1386.840334] LustreError: 52955:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1387.896975] Lustre: Modifying parameter general.timeout=1 in log params [ 1389.037816] Lustre: Modifying parameter general.timeout=2 in log params [ 1391.313760] Lustre: Modifying parameter general.timeout=4 in log params [ 1391.318255] Lustre: Skipped 1 previous similar message [ 1395.864884] Lustre: Modifying parameter general.timeout=9 in log params [ 1395.871288] Lustre: Skipped 4 previous similar messages [ 1404.415934] Lustre: Modifying parameter general.timeout=18 in log params [ 1404.420188] Lustre: Skipped 8 previous similar messages [ 1420.820404] Lustre: Modifying parameter general.timeout=36 in log params [ 1420.832644] Lustre: Skipped 17 previous similar messages [ 1453.003770] Lustre: Modifying parameter general.timeout=67 in log params [ 1453.007620] Lustre: Skipped 30 previous similar messages [ 1528.256219] Lustre: Disabling parameter general.timeout= in log params [ 1528.258781] Lustre: Skipped 62 previous similar messages [ 1532.835917] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:56:10 (1787381770) [ 1536.141415] Lustre: 59906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1538.503804] Lustre: 60016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1540.723288] Lustre: 60114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1545.157420] Lustre: 60311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1545.165895] Lustre: 60311:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1576.269722] Lustre: 61787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1576.276599] Lustre: 61787:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1576.305050] LustreError: 61787:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1581.613395] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:56:59 (1787381819) [ 1599.977701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.986311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.008730] Lustre: Skipped 2 previous similar messages [ 1600.027327] Lustre: Skipped 2 previous similar messages [ 1603.574996] Lustre: server umount lustre-MDT0000 complete [ 1605.089089] LustreError: 52268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.132994] LustreError: 52268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1607.765435] LustreError: 50406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381848 with bad export cookie 4926864653643031937 [ 1607.774608] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1607.799892] LustreError: 50406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1608.319499] Lustre: server umount lustre-MDT0001 complete [ 1622.763424] Lustre: server umount lustre-OST0000 complete [ 1651.798392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.097110] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1652.474413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.132352] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1666.542909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.725697] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1666.762704] Lustre: 64323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1671.403992] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1679.896711] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1686.244948] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1693.123363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1693.324676] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1693.595131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.601866] Lustre: Skipped 1 previous similar message [ 1699.387623] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1707.034902] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1708.858994] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1708.869800] Lustre: Skipped 23 previous similar messages [ 1708.929523] Lustre: 66888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1716.709181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1716.714276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.722135] Lustre: Skipped 1 previous similar message [ 1716.732912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1722.780508] Lustre: server umount lustre-OST0000 complete [ 1727.468141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.482446] Lustre: Skipped 2 previous similar messages [ 1741.279157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1741.483357] Lustre: server umount lustre-MDT0000 complete [ 1742.818229] LustreError: 64335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.843347] LustreError: 64335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1745.612194] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381986 with bad export cookie 4926864653643035122 [ 1745.614221] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.629900] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1746.216822] Lustre: server umount lustre-MDT0001 complete [ 1759.246979] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 1762.111261] Key type lgssc unregistered [ 1762.442599] LNet: 68102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.454941] LNetError: 68102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.477380] LNet: Removed LNI 192.168.204.151@tcp [ 1763.610726] Key type .llcrypt unregistered [ 1763.612358] Key type ._llcrypt unregistered [ 1795.876431] Key type ._llcrypt registered [ 1795.880673] Key type .llcrypt registered [ 1795.982174] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 1797.159149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.204681] alg: No test for adler32 (adler32-zlib) [ 1798.235700] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1798.480187] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 1800.151162] Key type lgssc registered [ 1801.367633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1810.849189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1810.877835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.400697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.114700] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1822.698281] LustreError: 69519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.719528] LustreError: 69519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1827.826599] LustreError: 69520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.677774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.443588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.043536] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1844.470661] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1851.374803] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1859.874282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1860.457773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.562422] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1876.115074] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1894.989187] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1906.149189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1906.162960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.191060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.714055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.718973] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.729492] Lustre: Skipped 1 previous similar message [ 1908.764895] Lustre: Skipped 1 previous similar message [ 1912.384934] Lustre: server umount lustre-MDT0000 complete [ 1913.827407] LustreError: 69519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.845640] LustreError: 69519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1916.476796] LustreError: 69501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382157 with bad export cookie 9097667349643251971 [ 1916.485983] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.502570] LustreError: 69501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1916.987256] Lustre: server umount lustre-MDT0001 complete [ 1932.259441] Lustre: server umount lustre-OST0000 complete [ 1965.072195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.382612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1965.419645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1965.708967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.919286] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1983.067847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.201169] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1983.256460] Lustre: 74429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1983.699663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.396378] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1998.513237] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2005.821163] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2013.882248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2014.086665] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2014.527535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.508304] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2029.939123] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2032.082686] Lustre: 77005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2034.078662] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2040.802962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.817552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.829055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.837981] Lustre: Skipped 1 previous similar message [ 2046.437478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2046.446639] Lustre: Skipped 2 previous similar messages [ 2046.725840] Lustre: server umount lustre-OST0000 complete [ 2051.553759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.560828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2051.573019] Lustre: Skipped 2 previous similar messages [ 2051.576086] Lustre: Skipped 1 previous similar message [ 2056.871629] Lustre: server umount lustre-MDT0000 complete [ 2060.941100] LustreError: 74422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382301 with bad export cookie 9097667349643252825 [ 2060.949740] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2060.959619] LustreError: 74422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2061.329344] Lustre: server umount lustre-MDT0001 complete [ 2073.869951] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 2077.156968] Key type lgssc unregistered [ 2077.610880] LNet: 78219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.633182] LNetError: 78219:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2078.699223] LNet: Removed LNI 192.168.204.151@tcp [ 2080.189988] Key type .llcrypt unregistered [ 2080.193660] Key type ._llcrypt unregistered [ 2107.335628] Key type ._llcrypt registered [ 2107.338507] Key type .llcrypt registered [ 2107.481055] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2108.471542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.619587] alg: No test for adler32 (adler32-zlib) [ 2109.782413] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2109.992249] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2111.687882] Key type lgssc registered [ 2112.838182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2122.037945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2122.070218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.800518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.336930] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2133.990317] LustreError: 79631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.006238] LustreError: 79631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2139.110234] LustreError: 79632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.502033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.331857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2148.544656] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2157.033080] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2164.082566] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2172.767913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2173.533340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.047200] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2189.478474] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2199.863417] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2200.845376] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2202.871128] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2202.875386] Lustre: Skipped 1 previous similar message [ 2206.025213] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2206.034523] Lustre: Skipped 2 previous similar messages [ 2210.878528] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2210.883957] Lustre: Skipped 3 previous similar messages [ 2219.701214] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2219.705753] Lustre: Skipped 7 previous similar messages [ 2236.174675] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2236.179418] Lustre: Skipped 14 previous similar messages [ 2268.356831] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2268.360843] Lustre: Skipped 31 previous similar messages [ 2318.169798] Lustre: 87748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2320.442742] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:09:19 (1787382559) [ 2321.774512] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2324.105457] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:09:22 (1787382562) [ 2328.554908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.556378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2328.576121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2328.576386] Lustre: Skipped 1 previous similar message [ 2333.664504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.672364] Lustre: Skipped 2 previous similar messages [ 2334.508335] Lustre: server umount lustre-OST0000 complete [ 2342.880676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2342.895715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.922508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.909228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.571659] Lustre: server umount lustre-MDT0000 complete [ 2349.025340] LustreError: 79627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.042705] LustreError: 79627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2349.541987] LustreError: 79614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382590 with bad export cookie 738042576454249575 [ 2349.552783] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2349.560528] LustreError: 79614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2349.973887] Lustre: server umount lustre-MDT0001 complete [ 2363.701960] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 2368.282973] Key type lgssc unregistered [ 2368.844910] LNet: 88998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.849984] LNetError: 88998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.890978] LNet: Removed LNI 192.168.204.151@tcp [ 2370.365966] Key type .llcrypt unregistered [ 2370.373646] Key type ._llcrypt unregistered [ 2375.950434] Key type ._llcrypt registered [ 2375.953159] Key type .llcrypt registered [ 2376.095611] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2381.747461] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules [ 2382.981583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2383.039094] alg: No test for adler32 (adler32-zlib) [ 2384.279867] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2384.784653] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2386.594089] Key type lgssc registered [ 2388.677518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2390.838995] LustreError: 90259:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2411.047873] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2414.253527] LustreError: 90259:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2420.252691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2420.276366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.893786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.807159] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2431.463861] LustreError: 91258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.480589] LustreError: 91258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2436.597892] LustreError: 91259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.806509] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:11:14 (1787382674) [ 2441.706470] LustreError: 91258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.880330] LustreError: 91290:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2442.907875] LustreError: 91290:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2442.923927] LustreError: 91290:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2443.419164] Lustre: server umount lustre-MDT0000 complete [ 2460.888437] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 2463.847024] Key type lgssc unregistered [ 2464.181639] LNet: 92784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.191500] LNetError: 92784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2465.254693] LNet: Removed LNI 192.168.204.151@tcp [ 2466.188137] Key type .llcrypt unregistered [ 2466.192781] Key type ._llcrypt unregistered [ 2485.718605] Key type ._llcrypt registered [ 2485.720985] Key type .llcrypt registered [ 2485.870543] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2486.797392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2486.939857] alg: No test for adler32 (adler32-zlib) [ 2488.044912] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2488.375023] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2490.039209] Key type lgssc registered [ 2491.283750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2499.308419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2499.327258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.900552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.432456] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2510.310360] LustreError: 94013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.334061] LustreError: 94013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2515.428738] LustreError: 94012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.673088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.250171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2521.257910] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2529.983432] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2536.623993] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2544.600181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2545.247933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.175954] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2559.031456] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2610.781316] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:14:08 (1787382848) [ 2613.109765] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2615.128447] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:14:13 (1787382853) [ 2622.434960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.453943] Lustre: Skipped 1 previous similar message [ 2622.458128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2622.483507] Lustre: Skipped 2 previous similar messages [ 2626.487222] Lustre: server umount lustre-MDT0000 complete [ 2627.555427] LustreError: 94008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.583183] LustreError: 94008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2630.619776] LustreError: 93993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382871 with bad export cookie 13960053630047188950 [ 2630.632452] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.646057] LustreError: 93993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2631.054757] Lustre: server umount lustre-MDT0001 complete [ 2646.332773] Lustre: server umount lustre-OST0000 complete [ 2660.906076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.386172] LustreError: 98309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.572166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.189119] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2666.977170] LustreError: 98310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.288113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.212358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.179796] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2689.231952] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2696.535366] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2711.180058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.380932] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2711.397334] LustreError: 98297:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2711.446898] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2711.454359] LustreError: 100394:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2711.463153] LustreError: 100394:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2711.468746] LustreError: 100394:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2711.554888] Lustre: server umount lustre-OST0000 complete [ 2711.563661] LustreError: 100394:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2720.750336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.971782] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2720.981652] LustreError: 98299:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2721.007953] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2721.017778] LustreError: 100861:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2721.029506] LustreError: 100861:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2721.038240] LustreError: 100861:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2721.098498] Lustre: server umount lustre-OST0000 complete [ 2721.103156] LustreError: 100861:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2731.538308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.805914] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2731.813349] Lustre: Found index 0 for lustre-OST0000, updating log [ 2731.820234] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2731.942313] Lustre: lustre-OST0000: new disk, initializing [ 2731.947476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2731.961778] Lustre: 101396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2732.250335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.158538] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2741.865068] Lustre: Failing over lustre-OST0000 [ 2742.028812] Lustre: server umount lustre-OST0000 complete [ 2746.592830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.608529] Lustre: Skipped 1 previous similar message [ 2746.614955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2746.620718] Lustre: Skipped 1 previous similar message [ 2751.968886] LustreError: 98309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.982933] LustreError: 98309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2753.868820] Lustre: server umount lustre-MDT0000 complete [ 2758.387270] LustreError: 98290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382999 with bad export cookie 13960053630047191694 [ 2758.388166] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2758.399034] LustreError: 98290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2758.824383] Lustre: server umount lustre-MDT0001 complete [ 2768.765288] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 03:16:46 (1787383006) [ 2788.805143] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2802.779724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.272859] LustreError: 103743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.289541] LustreError: 103743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2803.461139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.602480] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2820.917326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.374407] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2829.566909] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2836.814665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.611728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.616795] Lustre: Skipped 1 previous similar message [ 2844.374925] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2852.705158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.802833] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2852.908651] Lustre: lustre-OST0001: new disk, initializing [ 2852.918598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2852.928332] Lustre: 106064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2859.522351] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2859.617780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2859.638051] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2859.757974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2868.439418] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2876.967985] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2882.978505] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [ 2883.552940] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2883.562541] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.574277] Lustre: Skipped 1 previous similar message [ 2885.604839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2885.612239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.621793] Lustre: Skipped 1 previous similar message [ 2885.643679] Lustre: Skipped 1 previous similar message [ 2890.721644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2890.727090] Lustre: Skipped 3 previous similar messages [ 2896.351406] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2896.605620] Lustre: server umount lustre-MDT0001 complete [ 2898.296934] LustreError: 103737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.318706] LustreError: 103737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2907.263311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.397839] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2907.403601] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2907.458411] Lustre: 103733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2908.120111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2908.128754] Lustre: Skipped 1 previous similar message [ 2908.736023] LustreError: 107608:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2908.744796] LustreError: 107608:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2908.755768] LustreError: 103735:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2908.903222] LustreError: 107618:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.151@tcp: cfg command failed: rc = -17 [ 2908.930057] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2908.930057] [ 2912.868189] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2913.269596] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2924.225724] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 03:19:22 (1787383162) [ 2946.336798] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2965.173739] Lustre: 109679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2988.011461] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2992.254796] Lustre: 110814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3081.703838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3081.742837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3081.769441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.782264] Lustre: Skipped 5 previous similar messages [ 3084.164958] Lustre: server umount lustre-MDT0000 complete [ 3087.328833] LustreError: 103739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3087.373191] LustreError: 103739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3088.600164] LustreError: 103724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383329 with bad export cookie 13960053630047192688 [ 3088.605282] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3088.632751] LustreError: 103724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3088.932884] Lustre: server umount lustre-MDT0001 complete [ 3102.442480] Lustre: server umount lustre-OST0000 complete [ 3115.796660] Lustre: server umount lustre-OST0001 complete [ 3122.133257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3133.155709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3143.622360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3155.698566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3181.488328] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3191.431942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.465201] Lustre: lustre-MDT0000: reset Object Index mappings [ 3191.899185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.048440] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3204.995699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.015574] Lustre: lustre-MDT0001: reset Object Index mappings [ 3210.940766] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3214.455276] Lustre: 115920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3222.289406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.445476] LustreError: 116274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.473524] LustreError: 116274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3230.486388] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3230.983257] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3236.852479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3239.697487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.098370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3246.108895] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3247.427675] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3255.853232] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3260.288870] Lustre: 117796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3321.638421] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:25:59 (1787383559) [ 3348.452795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.469203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3348.491943] Lustre: Skipped 5 previous similar messages [ 3348.505211] Lustre: Skipped 6 previous similar messages [ 3352.569471] Lustre: server umount lustre-MDT0000 complete [ 3356.590776] LustreError: 114751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383597 with bad export cookie 13960053630047239581 [ 3356.599777] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3356.614720] LustreError: 114751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3363.384928] Lustre: server umount lustre-MDT0001 complete [ 3373.720604] Lustre: server umount lustre-OST0000 complete [ 3384.868665] Lustre: server umount lustre-OST0001 complete [ 3394.062765] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 3406.379955] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3447.939574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.264522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3448.287099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3448.382321] Lustre: lustre-MDT0000: new disk, initializing [ 3448.478639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.484941] Lustre: Skipped 3 previous similar messages [ 3448.500574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3453.346765] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3460.680033] Lustre: server umount lustre-MDT0000 complete [ 3471.053458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.415266] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3471.425635] Lustre: 122068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3471.435216] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3476.245918] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3481.738974] Lustre: server umount lustre-MDT0000 complete [ 3490.620932] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:28:48 (1787383728) [ 3492.298185] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3494.252950] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:28:52 (1787383732) [ 3512.831230] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3524.663770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.488876] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3540.647603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.783976] Lustre: 124116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3540.840633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3540.844117] Lustre: Skipped 1 previous similar message [ 3540.898733] Lustre: lustre-MDT0001: new disk, initializing [ 3541.052806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3541.066403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3546.339215] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3551.599896] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3558.064712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3558.387798] Lustre: lustre-OST0000: new disk, initializing [ 3558.392135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3558.404570] Lustre: 125696:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3559.585859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3559.593473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3559.709719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3564.656847] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3576.051391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.267312] Lustre: lustre-OST0001: new disk, initializing [ 3576.274350] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3576.285865] Lustre: 126566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3577.878300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3577.889805] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3578.067421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3583.359565] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3593.718787] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3597.900486] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4032.654552] Lustre: 132993:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4032.664527] Lustre: 132993:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4032.675695] Lustre: 132993:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4032.685844] Lustre: 132993:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4032.697355] Lustre: 132993:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4032.701641] Lustre: 132993:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4368.343456] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:43:25 (1787384605) [ 4385.767132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4385.778165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.794724] Lustre: Skipped 3 previous similar messages [ 4385.830545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4385.846883] Lustre: Skipped 4 previous similar messages [ 4391.598801] Lustre: server umount lustre-MDT0000 complete [ 4392.417047] LustreError: 124128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.451414] LustreError: 124128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4395.649353] LustreError: 124110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384636 with bad export cookie 13960053630047310106 [ 4395.667896] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.668563] LustreError: 124110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4397.541074] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.559775] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4397.566679] Lustre: Skipped 4 previous similar messages [ 4397.579784] Lustre: Skipped 5 previous similar messages [ 4402.452338] Lustre: server umount lustre-MDT0001 complete [ 4412.914533] Lustre: server umount lustre-OST0000 complete [ 4433.375783] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4433.816627] Lustre: server umount lustre-OST0001 complete [ 4442.471793] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 4454.387818] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 4496.470950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.750278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4496.802909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4496.906406] Lustre: lustre-MDT0000: new disk, initializing [ 4497.001094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.011424] Lustre: Skipped 5 previous similar messages [ 4497.031887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4502.915686] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4515.533707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4515.689842] Lustre: 136057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4515.795198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4515.803966] Lustre: Skipped 1 previous similar message [ 4515.947185] Lustre: lustre-MDT0001: new disk, initializing [ 4516.126593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4516.142744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4520.605425] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4531.446396] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4538.174377] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4545.023016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4545.471523] Lustre: lustre-OST0000: new disk, initializing [ 4545.476446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4545.488766] Lustre: 138027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4547.145488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4547.154414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4547.257423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4552.156933] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4562.623357] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4574.687308] Lustre: lustre-MDD0000: changelog on [ 4577.665667] Lustre: lustre-MDD0001: changelog on [ 6676.376360] Lustre: lustre-MDD0001: changelog off [ 6678.602744] Lustre: 136064:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6679.553939] Lustre: lustre-MDD0000: changelog off [ 6687.734924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6687.745765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6687.782074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6687.789315] Lustre: Skipped 2 previous similar messages [ 6690.805274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.824413] Lustre: Skipped 2 previous similar messages [ 6692.743665] Lustre: server umount lustre-OST0000 complete [ 6697.446564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6697.460866] LustreError: Skipped 1 previous similar message [ 6697.464166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.481078] Lustre: Skipped 1 previous similar message [ 6697.487558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6701.024917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6703.818732] Lustre: server umount lustre-MDT0000 complete [ 6706.145526] LustreError: 138592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6706.174946] LustreError: 138592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6708.274372] LustreError: 136049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386948 with bad export cookie 13960053630048133299 [ 6708.291309] LustreError: 136049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6708.294640] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6708.758869] Lustre: server umount lustre-MDT0001 complete [ 6721.790857] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 6724.752193] Key type lgssc unregistered [ 6725.018180] LNet: 147199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6725.024375] LNetError: 147199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6725.044687] LNet: Removed LNI 192.168.204.151@tcp [ 6725.989181] Key type .llcrypt unregistered [ 6725.991670] Key type ._llcrypt unregistered [ 6748.638780] Key type ._llcrypt registered [ 6748.642230] Key type .llcrypt registered [ 6748.786065] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6764.893084] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6765.930419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6766.053632] alg: No test for adler32 (adler32-zlib) [ 6767.147102] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 6767.431376] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 6769.073869] Key type lgssc registered [ 6770.248775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6808.230785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6808.268771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6809.538441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6809.586759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6809.718193] Lustre: lustre-MDT0000: new disk, initializing [ 6809.811409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6809.831808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6814.621988] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6827.595516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.753853] Lustre: 150402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6827.816318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6827.820066] Lustre: Skipped 1 previous similar message [ 6827.884739] Lustre: lustre-MDT0001: new disk, initializing [ 6828.042355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6828.092096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6828.104418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6833.223787] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6843.009537] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6848.774281] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6855.806960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.019301] Lustre: lustre-OST0000: new disk, initializing [ 6856.023664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6856.029574] Lustre: 152370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6856.119989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6863.490362] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6863.969331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6863.984533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6864.139642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6874.334800] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6880.414220] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_module obdecho/obdecho [ 6882.508270] LustreError: 153373:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6882.517941] LustreError: 153373:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6884.459851] LustreError: 153470:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6899.683961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6899.694312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.724494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6900.201161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.213924] Lustre: Skipped 1 previous similar message [ 6905.315257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.329461] Lustre: Skipped 5 previous similar messages [ 6905.904628] Lustre: server umount lustre-MDT0000 complete [ 6910.419955] LustreError: 150396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387151 with bad export cookie 15194312108628067269 [ 6910.429272] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6910.445618] LustreError: 150396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6910.493615] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6910.508548] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6911.130750] Lustre: server umount lustre-MDT0001 complete [ 6917.735735] Lustre: server umount lustre-OST0000 complete [ 6942.256788] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6957.811517] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7004.635903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.088546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7005.117899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7005.208607] Lustre: lustre-MDT0000: new disk, initializing [ 7005.333362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7005.357657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7010.915545] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7023.981554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.062584] Lustre: 157209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7024.107487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7024.111503] Lustre: Skipped 1 previous similar message [ 7024.175867] Lustre: lustre-MDT0001: new disk, initializing [ 7024.285881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7024.340816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7024.366469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7029.302598] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7041.081985] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7046.967037] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7053.861045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7054.147201] Lustre: lustre-OST0000: new disk, initializing [ 7054.150978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7054.156640] Lustre: 159176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7054.246889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7060.447698] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7063.640156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7063.656170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7063.713995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7070.222593] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7076.014432] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7076.244867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7081.526035] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7081.796328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7084.515178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7084.519418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7084.531177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7084.551189] Lustre: Skipped 1 previous similar message [ 7089.635206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7089.653476] Lustre: Skipped 2 previous similar messages [ 7098.335124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7098.540277] Lustre: server umount lustre-OST0000 complete [ 7104.993140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7105.029047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7105.046611] Lustre: Skipped 3 previous similar messages [ 7109.847715] Lustre: server umount lustre-MDT0000 complete [ 7110.121031] LustreError: 158433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7110.153434] LustreError: 158433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7114.578857] LustreError: 157201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387355 with bad export cookie 15194312108628068228 [ 7114.587845] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7114.589296] LustreError: 157201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7114.988790] Lustre: server umount lustre-MDT0001 complete [ 7119.001662] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 04:29:17 (1787387357) [ 7121.008683] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7123.226799] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 04:29:21 (1787387361) [ 7132.007648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.626773] LustreError: 161592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.731597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7137.768026] LustreError: 161593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.049643] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7142.881568] LustreError: 161592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.005497] LustreError: 161593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7149.823328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.040859] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7165.144232] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7171.602830] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7179.600988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7179.967517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7179.971020] Lustre: Skipped 1 previous similar message [ 7187.151597] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7196.593916] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7212.001150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7212.015941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.041828] Lustre: Skipped 1 previous similar message [ 7212.067534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7214.412388] Lustre: server umount lustre-MDT0000 complete [ 7218.040155] LustreError: 162883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.574287] LustreError: 161573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387459 with bad export cookie 15194312108628069110 [ 7218.575623] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7218.596852] LustreError: 161573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7225.075413] Lustre: server umount lustre-MDT0001 complete [ 7233.266919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7249.782230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7250.063428] LustreError: 165440:0:(llog.c:1655:llog_backup()) MGC192.168.204.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7250.075087] Lustre: 165440:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7253.753828] LustreError: 165445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7253.778212] LustreError: 165445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7288.694602] LustreError: 165446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.719461] LustreError: 165446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 7310.815320] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7310.821979] Lustre: 165440:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7351.802215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7353.209633] LustreError: 165446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.251387] LustreError: 165446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 7356.909941] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7357.634415] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7368.617529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.436507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7388.647163] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd2dd0a5c2131644f [ 7388.667528] Lustre: MGC192.168.204.151@tcp: Connection restored to 0@lo (at 0@lo) [ 7394.311699] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7394.723376] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7403.804065] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7406.039861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7412.064778] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7414.413126] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7438.308168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.310707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.349475] Lustre: Skipped 5 previous similar messages [ 7438.365667] Lustre: Skipped 8 previous similar messages [ 7443.894910] Lustre: server umount lustre-MDT0000 complete [ 7449.545185] LustreError: 161573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387690 with bad export cookie 15194312108628075599 [ 7449.559976] LustreError: 161573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7449.561813] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.150901] Lustre: server umount lustre-MDT0001 complete [ 7464.727544] Lustre: server umount lustre-OST0000 complete [ 7491.530447] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 7507.125992] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7552.595946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7552.772511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7552.814804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7552.951495] Lustre: lustre-MDT0000: new disk, initializing [ 7553.074693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7553.088189] Lustre: Skipped 1 previous similar message [ 7553.139948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7558.878542] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7570.672180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.795698] Lustre: 170579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7571.026471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7576.230617] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7588.385701] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7596.623616] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7605.758246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7606.431464] Lustre: lustre-OST0000: new disk, initializing [ 7606.441441] Lustre: Skipped 1 previous similar message [ 7606.450715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7606.457855] Lustre: Skipped 2 previous similar messages [ 7606.464095] Lustre: 172547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7615.634960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7615.651256] Lustre: Skipped 1 previous similar message [ 7615.658942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7615.756593] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7615.773491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7627.679411] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7634.047651] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7634.334155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7640.096811] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7640.340985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7646.701069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.705118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7646.721177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7646.721189] Lustre: Skipped 3 previous similar messages [ 7646.721221] Lustre: Skipped 1 previous similar message [ 7649.145712] Lustre: server umount lustre-OST0000 complete [ 7659.982029] Lustre: server umount lustre-MDT0000 complete [ 7662.061137] LustreError: 170589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7662.082103] LustreError: 170589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 7664.089891] LustreError: 172546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387904 with bad export cookie 15194312108628092413 [ 7664.095568] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7664.101090] LustreError: 172546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7664.419519] Lustre: server umount lustre-MDT0001 complete [ 7674.511479] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 04:38:32 (1787387912) [ 7681.712535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.512144] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7695.609642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.397620] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7708.927129] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7715.117533] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7722.547437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7729.010233] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7736.832443] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7752.352599] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 04:39:50 (1787387990) [ 7757.793972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7757.807959] LustreError: Skipped 1 previous similar message [ 7757.814929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.834313] Lustre: Skipped 2 previous similar messages [ 7763.295722] Lustre: server umount lustre-OST0000 complete [ 7773.950614] Lustre: server umount lustre-MDT0000 complete [ 7778.014078] LustreError: 174941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388018 with bad export cookie 15194312108628093295 [ 7778.016168] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7778.031719] LustreError: 174941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7778.350116] Lustre: server umount lustre-MDT0001 complete [ 7789.097861] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7791.852369] Key type lgssc unregistered [ 7792.215740] LNet: 178691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7792.236179] LNetError: 178691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7793.255979] LNet: Removed LNI 192.168.204.151@tcp [ 7794.309053] Key type .llcrypt unregistered [ 7794.317393] Key type ._llcrypt unregistered [ 7814.132389] Key type ._llcrypt registered [ 7814.135474] Key type .llcrypt registered [ 7814.220941] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7815.410987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7815.564839] alg: No test for adler32 (adler32-zlib) [ 7816.697207] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 7816.971658] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 7818.687224] Key type lgssc registered [ 7819.820753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7828.115140] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7828.144263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.743389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7848.863465] LustreError: 179936:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.151@tcp: failed processing log, type 4: rc = -110 [ 7853.983515] LustreError: 179936:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7854.003221] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7854.013104] LustreError: MGC192.168.204.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7854.030568] LustreError: 179936:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7854.039895] LustreError: 179936:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7854.048382] LustreError: 179936:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7854.153204] Lustre: server umount lustre-OST0000 complete [ 7854.155554] LustreError: 179936:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7863.638071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7865.085479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7869.971774] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7874.530921] LustreError: 180436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.542151] LustreError: 180436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7878.799118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7879.129165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7884.104152] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7891.705575] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7897.431302] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7904.595666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.918369] LustreError: 182253:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7904.943732] Lustre: 182253:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7905.114459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7911.791953] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7919.973255] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7924.256975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.262317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7924.298455] Lustre: Skipped 1 previous similar message [ 7924.327592] Lustre: Skipped 2 previous similar messages [ 7928.496121] Lustre: server umount lustre-MDT0000 complete [ 7929.315546] LustreError: 180436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7929.331224] LustreError: 180436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7931.992047] LustreError: 180416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388172 with bad export cookie 2680229766858359492 [ 7932.006307] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7932.007736] LustreError: 180416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7932.471431] LustreError: 181164:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000005169268 x1874212219285504/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7932.649363] Lustre: server umount lustre-MDT0001 complete [ 7946.890811] Lustre: server umount lustre-OST0000 complete [ 7958.398903] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 7969.210522] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8009.914722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.334824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8010.367524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8010.464966] Lustre: lustre-MDT0000: new disk, initializing [ 8010.540134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8010.575961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8014.538404] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8025.040978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8025.148395] Lustre: 185780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8025.192746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8025.196528] Lustre: Skipped 1 previous similar message [ 8025.249741] Lustre: lustre-MDT0001: new disk, initializing [ 8025.322582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8025.351510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8025.364865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8029.406932] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8038.570109] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8044.543648] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8051.088689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8051.353134] Lustre: lustre-OST0000: new disk, initializing [ 8051.360455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8051.368655] Lustre: 187748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8051.465422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.459193] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8061.003855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8061.020481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8061.092738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8066.411356] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8071.699317] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8072.017455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8076.217312] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8076.430478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8081.888356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8081.899699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.907333] LustreError: Skipped 1 previous similar message [ 8081.910265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8081.940194] Lustre: Skipped 2 previous similar messages [ 8084.724910] Lustre: server umount lustre-OST0000 complete [ 8092.127980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8092.131477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8092.139521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.139529] Lustre: Skipped 1 previous similar message [ 8092.165934] Lustre: Skipped 1 previous similar message [ 8097.253935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8097.265878] Lustre: Skipped 2 previous similar messages [ 8101.345409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.361830] Lustre: Skipped 1 previous similar message [ 8103.391230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8103.632202] Lustre: server umount lustre-MDT0000 complete [ 8106.465891] LustreError: 188046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.492062] LustreError: 188046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8107.906524] LustreError: 187747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388348 with bad export cookie 2680229766858360213 [ 8107.925909] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8107.933524] LustreError: 187747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8108.383799] Lustre: server umount lustre-MDT0001 complete [ 8118.550189] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 04:45:56 (1787388356) [ 8137.348341] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 8140.487140] Key type lgssc unregistered [ 8140.814433] LNet: 190693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8140.823441] LNetError: 190693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8140.835667] LNet: Removed LNI 192.168.204.151@tcp [ 8141.855537] Key type .llcrypt unregistered [ 8141.857668] Key type ._llcrypt unregistered [ 8160.452185] Key type ._llcrypt registered [ 8160.455101] Key type .llcrypt registered [ 8160.581461] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8161.704925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8161.731693] alg: No test for adler32 (adler32-zlib) [ 8162.835773] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 8163.050980] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 8164.712498] Key type lgssc registered [ 8166.021572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8174.727401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8174.744386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.081208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8181.762824] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8185.313253] LustreError: 191913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8185.341108] LustreError: 191913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8190.433377] LustreError: 191912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.705837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8191.054959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8195.779635] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8204.973545] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8210.811953] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8218.555069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8218.757913] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8218.761240] LustreError: 193776:0:(llog.c:1696:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8218.769588] Lustre: 193776:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8218.903902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.692671] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8233.683485] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8237.025244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8237.035622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.055271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8240.610921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.617451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8240.653148] Lustre: Skipped 1 previous similar message [ 8240.671643] Lustre: Skipped 2 previous similar messages [ 8242.483743] Lustre: server umount lustre-MDT0000 complete [ 8245.740189] LustreError: 194197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8245.761054] LustreError: 194197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8246.460205] LustreError: 191895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388487 with bad export cookie 9882700254422674001 [ 8246.460731] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8246.476465] LustreError: 191895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8246.921783] Lustre: server umount lustre-MDT0001 complete [ 8261.393230] Lustre: server umount lustre-OST0000 complete [ 8271.522635] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8282.693869] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8328.460700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.877322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8328.930635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8329.097375] Lustre: lustre-MDT0000: new disk, initializing [ 8329.263463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.293524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8334.460291] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8346.547609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.722577] Lustre: 197304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8346.761400] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8346.763739] Lustre: Skipped 1 previous similar message [ 8346.766227] LustreError: 198088:0:(llog.c:1696:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8346.771363] LustreError: 198088:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8346.777805] Lustre: 198088:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8346.797356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8346.801297] Lustre: Skipped 1 previous similar message [ 8346.868826] Lustre: lustre-MDT0001: new disk, initializing [ 8346.995516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8347.053580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8347.061940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8351.755848] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8361.757555] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8367.989907] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8375.356486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8375.592784] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8375.596307] Lustre: Skipped 1 previous similar message [ 8375.601573] LustreError: 199250:0:(llog.c:1696:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8375.608291] LustreError: 199250:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8375.616583] Lustre: 199250:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8375.623903] Lustre: 199250:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8375.650337] Lustre: lustre-OST0000: new disk, initializing [ 8375.657612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8375.667179] Lustre: 199274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8375.783287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8382.574163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8382.585277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8382.720343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8382.876128] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8393.329358] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8398.801746] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8398.998821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8404.474621] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8404.808612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8408.548939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8408.554660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8408.571505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8408.587031] LustreError: Skipped 1 previous similar message [ 8413.666282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8413.680795] Lustre: Skipped 2 previous similar messages [ 8418.795647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8418.813491] Lustre: Skipped 1 previous similar message [ 8421.343582] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8421.672177] Lustre: server umount lustre-OST0000 complete [ 8429.024537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.049614] Lustre: Skipped 1 previous similar message [ 8429.052816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8429.063376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8432.876375] Lustre: server umount lustre-MDT0000 complete [ 8434.144744] LustreError: 197315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8434.177608] LustreError: 197315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8437.270431] LustreError: 197295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388677 with bad export cookie 9882700254422674715 [ 8437.280323] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8437.292861] LustreError: 197295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8437.697829] Lustre: server umount lustre-MDT0001 complete [ 8447.297640] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 04:51:25 (1787388685) [ 8466.682931] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8478.123953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8478.444380] LustreError: 202403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8478.497171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8483.565132] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8488.935025] LustreError: 202404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8488.949748] LustreError: 202404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8492.195560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.215757] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8500.058758] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8506.430076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8511.921410] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8519.539801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8519.721287] Lustre: lustre-OST0001: new disk, initializing [ 8519.724904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8519.730179] Lustre: 204713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8519.817941] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8519.826815] Lustre: Skipped 2 previous similar messages [ 8526.048523] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8526.413518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8526.427562] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8526.546159] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8536.879892] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8544.111561] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8546.263335] Lustre: DEBUG MARKER: ADD OST3 [ 8552.861022] loop: module loaded [ 8552.918156] loop0: detected capacity change from 0 to 204800000 [ 8553.069389] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8553.077978] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8553.086731] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8561.383818] Lustre: DEBUG MARKER: STOP OST3 [ 8561.878556] loop0: detected capacity change from 0 to 204800000 [ 8569.867834] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8570.094601] Lustre: lustre-OST0003: new disk, initializing [ 8570.098663] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8570.112749] Lustre: 206792:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8571.572654] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8571.761052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8571.769196] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8571.812084] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8572.707280] Lustre: Failing over lustre-OST0003 [ 8572.821718] Lustre: server umount lustre-OST0003 complete [ 8573.606535] LustreError: 206887:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8576.932725] LustreError: 203890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8576.994772] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8577.003410] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8577.019840] Lustre: Skipped 1 previous similar message [ 8578.448781] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8590.452482] Lustre: DEBUG MARKER: START OST3 again [ 8597.592700] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8597.960748] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8597.970292] Lustre: Skipped 1 previous similar message [ 8597.985273] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8599.026899] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8599.356173] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8599.357528] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8605.153298] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8629.630693] Lustre: Failing over lustre-OST0003 [ 8629.708385] Lustre: server umount lustre-OST0003 complete [ 8630.241140] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8630.249780] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8630.257549] Lustre: Skipped 1 previous similar message [ 8630.272919] LustreError: 203888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8630.284769] LustreError: 203888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 8649.207448] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.210857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8649.223239] Lustre: Skipped 2 previous similar messages [ 8649.251988] Lustre: Skipped 4 previous similar messages [ 8654.704402] Lustre: server umount lustre-MDT0000 complete [ 8658.790047] LustreError: 204712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388899 with bad export cookie 9882700254422675597 [ 8658.796979] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8658.812080] LustreError: 204712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8673.247170] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8673.742894] Lustre: server umount lustre-MDT0001 complete [ 8678.428868] Lustre: server umount lustre-OST0000 complete [ 8682.883519] Lustre: server umount lustre-OST0001 complete [ 8690.688704] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8701.451964] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8750.836805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.127532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8751.170698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8751.257107] Lustre: lustre-MDT0000: new disk, initializing [ 8751.365234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8751.408340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8756.938193] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8770.478194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8770.693860] Lustre: 212335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8770.785413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8770.789769] Lustre: Skipped 1 previous similar message [ 8770.944682] Lustre: lustre-MDT0001: new disk, initializing [ 8771.261218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8771.287071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8778.092340] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8789.325906] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8795.381498] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8803.191450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8803.436984] Lustre: lustre-OST0000: new disk, initializing [ 8803.444728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8803.457477] Lustre: 214304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8805.305127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8805.324566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8805.530337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8810.192576] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8820.620280] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8826.345661] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8826.563139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8832.039884] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8832.223708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8836.074343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8836.075681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.081215] LustreError: Skipped 2 previous similar messages [ 8836.095396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8836.103328] Lustre: Skipped 5 previous similar messages [ 8836.126750] Lustre: Skipped 11 previous similar messages [ 8840.490084] Lustre: server umount lustre-OST0000 complete [ 8850.896122] Lustre: server umount lustre-MDT0000 complete [ 8851.424703] LustreError: 213552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8851.444684] LustreError: 213552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 8854.859210] LustreError: 212327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389095 with bad export cookie 9882700254422677900 [ 8854.874593] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8855.272245] Lustre: server umount lustre-MDT0001 complete [ 8859.182363] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 04:58:17 (1787389097) [ 8877.458459] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8889.848448] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8931.343619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8931.626717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8931.731789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8931.870361] Lustre: lustre-MDT0000: new disk, initializing [ 8932.031912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8937.682188] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8948.997956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8949.161770] Lustre: 218915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8949.557117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8954.834172] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8965.913545] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8973.116300] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8981.318447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8981.730641] Lustre: 220887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8983.487395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8983.639789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8989.220166] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8999.428827] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9004.964056] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9005.273332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9010.037424] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9010.232914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9014.249252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9014.250590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.260372] LustreError: Skipped 1 previous similar message [ 9014.261105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9014.282092] Lustre: Skipped 3 previous similar messages [ 9014.300575] Lustre: Skipped 3 previous similar messages [ 9018.260656] Lustre: server umount lustre-OST0000 complete [ 9028.748094] Lustre: server umount lustre-MDT0000 complete [ 9029.604223] LustreError: 218927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9029.619853] LustreError: 218927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9033.073287] LustreError: 218909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389273 with bad export cookie 9882700254422678782 [ 9033.085350] LustreError: 218909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9033.089665] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9033.478904] Lustre: server umount lustre-MDT0001 complete [ 9042.397249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.043494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.049719] Lustre: Skipped 5 previous similar messages [ 9049.139455] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9058.951170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.435487] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9072.042398] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9077.351752] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9083.894813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9090.207193] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9098.157645] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9109.438174] Lustre: server umount lustre-OST0000 complete [ 9123.368945] Lustre: server umount lustre-MDT0000 complete [ 9127.204413] LustreError: 223096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389367 with bad export cookie 9882700254422679664 [ 9127.215971] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9127.219901] LustreError: 223096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9136.778739] Lustre: server umount lustre-MDT0001 complete [ 9238.009200] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 9241.403808] Key type lgssc unregistered [ 9241.891300] LNet: 226779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9241.905283] LNetError: 226779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9241.947405] LNet: Removed LNI 192.168.204.151@tcp [ 9243.415773] Key type .llcrypt unregistered [ 9243.417754] Key type ._llcrypt unregistered [ 9276.288596] Key type ._llcrypt registered [ 9276.291318] Key type .llcrypt registered [ 9276.403959] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 9292.041602] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 9293.201359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9293.227175] alg: No test for adler32 (adler32-zlib) [ 9294.449751] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 9294.679836] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 9296.391384] Key type lgssc registered [ 9297.463422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9334.361778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9334.422327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.667543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9335.700984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9335.809721] Lustre: lustre-MDT0000: new disk, initializing [ 9335.885266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9335.902563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9340.315643] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9351.203644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.294480] Lustre: 230135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9351.337917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9351.342595] Lustre: Skipped 1 previous similar message [ 9351.414600] Lustre: lustre-MDT0001: new disk, initializing [ 9351.497731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9351.529373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9351.543895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9357.151515] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9367.436993] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9373.813154] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9381.253912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9381.550943] Lustre: lustre-OST0000: new disk, initializing [ 9381.554495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9381.562214] Lustre: 232104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9381.626228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.061848] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9389.141305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9389.151702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9389.275925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9397.752331] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9403.068802] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9403.301873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9407.607702] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9407.868765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9410.016976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9410.023161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9410.030312] LustreError: Skipped 1 previous similar message [ 9410.031366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9410.047565] Lustre: Skipped 1 previous similar message [ 9410.059719] Lustre: Skipped 1 previous similar message [ 9415.138803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9415.144789] Lustre: Skipped 1 previous similar message [ 9415.924891] Lustre: server umount lustre-OST0000 complete [ 9420.268698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9420.277652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9420.293870] Lustre: Skipped 1 previous similar message [ 9420.309701] Lustre: Skipped 1 previous similar message [ 9425.378603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9425.386603] Lustre: Skipped 1 previous similar message [ 9430.496328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9430.507829] Lustre: Skipped 1 previous similar message [ 9434.079228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9434.248849] Lustre: server umount lustre-MDT0000 complete [ 9437.982791] LustreError: 230128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389678 with bad export cookie 2793938462707824115 [ 9437.984452] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9438.002030] LustreError: 230128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9438.276547] Lustre: server umount lustre-MDT0001 complete [ 9458.175589] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 9469.271480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9469.793861] LustreError: 235038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.806764] LustreError: 235038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9469.852369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9474.573840] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9475.041558] LustreError: 235039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9480.161390] LustreError: 235038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9483.328444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.694336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9487.972193] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9490.862155] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9497.877052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9498.238967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9504.901891] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9513.772535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9513.932183] Lustre: lustre-OST0001: new disk, initializing [ 9513.935520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9513.941455] Lustre: 237344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9519.203397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9519.214505] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9519.334970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9520.612440] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9533.069592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9541.047928] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9556.310883] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:09:54 (1787389794) [ 9560.546793] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9560.548990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9560.579604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9560.601448] Lustre: Skipped 3 previous similar messages [ 9564.698159] Lustre: server umount lustre-MDT0000 complete [ 9565.668879] LustreError: 235038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9565.689408] LustreError: 235038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9568.584475] LustreError: 235019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389809 with bad export cookie 2793938462707824997 [ 9568.593028] LustreError: 235019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9568.599507] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9569.030875] Lustre: server umount lustre-MDT0001 complete [ 9583.745499] Lustre: server umount lustre-OST0000 complete [ 9586.975313] Lustre: 228360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389811/real 1787389811] req@ffffa065e07f0700 x1874213769164672/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787389827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9587.018330] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9587.032297] Lustre: Skipped 3 previous similar messages [ 9588.223634] Lustre: server umount lustre-OST0001 complete [ 9597.032377] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 9608.395476] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 9644.276653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.663498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9644.718374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9644.813525] Lustre: lustre-MDT0000: new disk, initializing [ 9644.870940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.882653] Lustre: Skipped 1 previous similar message [ 9644.898540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9650.015435] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9660.373904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.564487] Lustre: 241294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9660.628783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9660.633807] Lustre: Skipped 1 previous similar message [ 9660.765557] Lustre: lustre-MDT0001: new disk, initializing [ 9660.902645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9660.913472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9665.190758] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9675.293954] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9682.779603] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9692.045128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9692.472463] Lustre: lustre-OST0000: new disk, initializing [ 9692.479266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9692.487925] Lustre: 243261:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9692.668945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9692.679814] Lustre: Skipped 1 previous similar message [ 9694.374899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9694.379961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9694.490306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9702.071686] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9712.847912] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9717.587438] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9717.796817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9722.499066] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9722.778575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9725.409251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9725.413470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9725.421101] LustreError: Skipped 1 previous similar message [ 9725.426859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9725.441102] Lustre: Skipped 1 previous similar message [ 9731.459624] Lustre: server umount lustre-OST0000 complete [ 9737.696146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9737.700855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.894889] Lustre: server umount lustre-MDT0000 complete [ 9745.891191] LustreError: 242457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.926320] LustreError: 242457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9747.049921] LustreError: 241285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389987 with bad export cookie 2793938462707827132 [ 9747.068288] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9747.583567] Lustre: server umount lustre-MDT0001 complete [ 9756.509551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.946904] LustreError: 245478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.034775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9761.432565] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9769.410508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.778285] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9781.467658] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9787.090675] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9795.672087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9803.969832] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 9813.225395] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9957.857393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9957.869155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9957.877837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9957.892806] Lustre: Skipped 2 previous similar messages [ 9957.919685] Lustre: Skipped 7 previous similar messages [ 9972.191193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9972.462554] Lustre: server umount lustre-OST0000 complete [ 9982.465308] Lustre: server umount lustre-MDT0000 complete [ 9983.464078] LustreError: 247993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9983.482319] LustreError: 247993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9986.284594] LustreError: 245460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390227 with bad export cookie 2793938462707828007 [ 9986.293800] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9986.301402] LustreError: 245460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9986.630766] Lustre: server umount lustre-MDT0001 complete [10079.848613] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [10082.196913] Key type lgssc unregistered [10082.572895] LNet: 249083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10082.580820] LNetError: 249083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10082.598071] LNet: Removed LNI 192.168.204.151@tcp [10083.409272] Key type .llcrypt unregistered [10083.412438] Key type ._llcrypt unregistered [10111.395661] Key type ._llcrypt registered [10111.398819] Key type .llcrypt registered [10111.553556] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [10129.646245] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10130.929766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10131.024820] alg: No test for adler32 (adler32-zlib) [10132.199448] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [10132.564745] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [10134.343204] Key type lgssc registered [10135.235809] Lustre: Echo OBD driver; http://www.lustre.org/ [10184.755818] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10195.496843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10195.532353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.822756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10196.869614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10196.991869] Lustre: lustre-MDT0000: new disk, initializing [10197.100138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10197.144831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10201.954970] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10211.870970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10212.012270] Lustre: 253157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10212.066509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10212.070182] Lustre: Skipped 1 previous similar message [10212.179603] Lustre: lustre-MDT0001: new disk, initializing [10212.325095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10212.360894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10212.376514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10217.318148] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10222.309972] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10228.815954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.058902] Lustre: lustre-OST0000: new disk, initializing [10229.064779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10229.070449] Lustre: 254788:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10229.156449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10233.930124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10233.941229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10234.115975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10235.620654] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10249.146350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10249.461154] Lustre: lustre-OST0001: new disk, initializing [10249.474372] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10249.485589] Lustre: 255660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10249.755208] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10256.986571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10256.999169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10257.145030] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10258.915264] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10270.329677] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10278.107239] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10293.217949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10293.226833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10293.245190] Lustre: Skipped 2 previous similar messages [10293.263959] Lustre: Skipped 3 previous similar messages [10297.218934] Lustre: server umount lustre-MDT0000 complete [10301.088700] LustreError: 253148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390541 with bad export cookie 8603138198433977057 [10301.103179] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10301.104852] LustreError: 253148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10301.380072] Lustre: server umount lustre-MDT0001 complete [10315.850457] Lustre: server umount lustre-OST0000 complete [10319.839163] Lustre: 250663:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390544/real 1787390544] req@ffffa065c6190a80 x1874214647841024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10319.879746] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10319.899103] Lustre: Skipped 1 previous similar message [10320.184587] Lustre: server umount lustre-OST0001 complete [10335.024072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10347.208846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10357.702582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10367.694807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10390.309330] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10401.467184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10401.496955] Lustre: lustre-MDT0000: reset Object Index mappings [10401.919531] LustreError: 260672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.946694] LustreError: 260672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10402.030890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10406.901037] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10407.407543] LustreError: 260673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.431336] LustreError: 260673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10411.565962] LustreError: 260672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.612475] LustreError: 260673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.729732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10416.744588] Lustre: lustre-MDT0001: reset Object Index mappings [10417.246122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10422.549449] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10426.022905] Lustre: 261816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10434.634041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.050446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10442.225945] LustreError: 262169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10442.489690] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10446.342923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10451.440531] LustreError: 262172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.465145] LustreError: 262172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10451.923504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10457.605359] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10459.948922] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10468.604641] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10473.421344] Lustre: 263689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10479.257735] Lustre: server umount lustre-MDT0000 complete [10483.009503] LustreError: 260654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390723 with bad export cookie 8603138198433983847 [10483.029233] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10483.033250] LustreError: 260654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10483.174958] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10483.181769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10483.181881] LustreError: 260668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10483.191189] Lustre: Skipped 3 previous similar messages [10483.224618] Lustre: Skipped 1 previous similar message [10485.515318] Lustre: server umount lustre-MDT0001 complete [10489.494987] Lustre: server umount lustre-OST0000 complete [10493.642183] Lustre: server umount lustre-OST0001 complete [10515.111868] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 05:25:53 (1787390753) [10534.604469] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [10548.154860] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10585.308290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10585.534205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10585.555546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10585.613492] Lustre: lustre-MDT0000: new disk, initializing [10585.677281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10585.682870] Lustre: Skipped 1 previous similar message [10585.700246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10590.233892] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10601.022204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10601.164334] Lustre: 267943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10601.216716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10601.221556] Lustre: Skipped 1 previous similar message [10601.307636] Lustre: lustre-MDT0001: new disk, initializing [10601.425414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10601.441567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10606.062242] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10616.334354] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10623.421218] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10632.336568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10632.700072] Lustre: lustre-OST0000: new disk, initializing [10632.704716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10632.711748] Lustre: 269912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10634.157505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10634.175609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10634.308501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10639.099771] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10651.856413] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10658.643670] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10658.973190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10664.190284] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10664.471105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10670.052103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10670.052161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10670.078931] LustreError: Skipped 1 previous similar message [10670.081808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10670.090786] Lustre: Skipped 1 previous similar message [10670.112671] Lustre: Skipped 1 previous similar message [10672.534832] Lustre: server umount lustre-OST0000 complete [10678.241149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10678.251068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.281892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.262900] Lustre: server umount lustre-MDT0000 complete [10685.423028] LustreError: 267954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10685.448889] LustreError: 267954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10687.385577] LustreError: 268748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390928 with bad export cookie 8603138198433985940 [10687.394950] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10687.408257] LustreError: 268748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10687.977251] Lustre: server umount lustre-MDT0001 complete [10709.138791] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10720.341613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10720.820739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10720.825975] Lustre: Skipped 2 previous similar messages [10725.597268] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10735.110398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10741.016877] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10744.223282] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10752.377071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10763.171398] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10775.172397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10775.415841] Lustre: lustre-OST0001: new disk, initializing [10775.419527] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10775.425087] Lustre: 275160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10777.245607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10777.271474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10777.394639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10783.853520] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10794.397093] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10799.003725] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10804.554248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10804.631484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10812.384334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10812.398158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10812.420771] Lustre: Skipped 1 previous similar message [10812.450632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10812.459729] Lustre: Skipped 2 previous similar messages [10817.601341] Lustre: server umount lustre-MDT0000 complete [10818.534241] LustreError: 272847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.562957] LustreError: 272847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10821.502741] LustreError: 272828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391062 with bad export cookie 8603138198433986822 [10821.503965] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10821.529510] LustreError: 272828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10821.979577] Lustre: server umount lustre-MDT0001 complete [10836.359830] Lustre: server umount lustre-OST0000 complete [10839.520756] Lustre: 250664:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787391064/real 1787391064] req@ffffa065fe662300 x1874214648004736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787391080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10839.559443] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10839.578147] Lustre: Skipped 3 previous similar messages [10840.635178] Lustre: server umount lustre-OST0001 complete [10861.465750] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [10872.190372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10872.716134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10872.723292] Lustre: Skipped 3 previous similar messages [10878.614712] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10889.475336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.746226] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10897.675376] Lustre: 279357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10903.988127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10910.298630] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10917.769022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10919.424882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.080187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10926.302895] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [10933.853369] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10937.547416] Lustre: 281231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10943.372332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10943.417229] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10954.976848] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 05:33:12 (1787391192) [10971.107530] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.110095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.119653] Lustre: Skipped 1 previous similar message [10971.135722] Lustre: Skipped 7 previous similar messages [10977.046990] Lustre: server umount lustre-MDT0000 complete [10980.955794] LustreError: 278198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391221 with bad export cookie 8603138198433993360 [10980.965826] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10980.978522] LustreError: 278198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10981.345563] LustreError: 278211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.367933] LustreError: 278211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [10987.531427] Lustre: server umount lustre-MDT0001 complete [11005.407178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11005.531809] Lustre: server umount lustre-OST0000 complete [11008.993967] Lustre: server umount lustre-OST0001 complete [11015.860955] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [11027.496591] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [11074.297984] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [11085.228789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11085.517457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11085.581481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11085.690856] Lustre: lustre-MDT0000: new disk, initializing [11085.780785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11090.459440] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11101.128946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.283771] Lustre: 285470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11101.744828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11106.564415] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11118.247778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11118.551542] Lustre: 287104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11120.006556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11125.667651] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11136.769356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11137.011319] Lustre: 287973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11137.155372] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11137.159860] Lustre: Skipped 6 previous similar messages [11138.443037] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11138.455045] Lustre: Skipped 1 previous similar message [11138.579386] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11143.905301] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11154.661329] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11159.017093] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11159.026123] Lustre: Skipped 1 previous similar message [11176.986802] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 05:36:54 (1787391414) [11179.496441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11179.504274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11179.522594] Lustre: Skipped 6 previous similar messages [11179.526955] Lustre: Skipped 10 previous similar messages [11179.742236] Lustre: server umount lustre-MDT0000 complete [11183.681449] LustreError: 285464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391424 with bad export cookie 8603138198434001011 [11183.685498] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11183.698582] LustreError: 285464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11183.971630] Lustre: server umount lustre-MDT0001 complete [11198.467898] Lustre: server umount lustre-OST0000 complete [11200.738305] Lustre: 250666:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787391425/real 1787391425] req@ffffa064cf888000 x1874214648155648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787391441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11202.832872] Lustre: server umount lustre-OST0001 complete [11213.623080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.288833] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11230.049696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11235.438937] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11245.629110] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11253.394512] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11261.857018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11270.362268] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11276.838567] LustreError: 292050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.882252] LustreError: 292050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11279.766458] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11306.166487] loop1: detected capacity change from 0 to 204800000 [11306.517188] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11306.533984] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11306.549120] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11312.805464] loop1: detected capacity change from 0 to 204800000 [11319.132385] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11319.267846] LustreError: 290205:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13549: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11319.284496] LustreError: 290205:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13549 [11319.297112] LustreError: 290205:0:(mgs_handler.c:517:mgs_target_reg()) fs13549-MDT0001: Error checking target index: rc = -13 [11319.308979] LustreError: fs13549-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11319.319933] LustreError: 293942:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11319.327712] LustreError: 293942:0:(tgt_mount.c:2129:server_put_super()) no obd fs13549-MDT0001 [11319.333304] LustreError: 293942:0:(tgt_mount.c:116:server_deregister_mount()) fs13549-MDT0001 not registered [11319.369824] Lustre: server umount fs13549-MDT0001 complete [11319.372880] LustreError: 293942:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11329.014160] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11329.175359] Lustre: Setting parameter fs13549-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13549-MDT0001 [11329.263395] Lustre: srv-fs13549-MDT0001: No data found on store. Initialize space. [11329.272496] Lustre: Skipped 4 previous similar messages [11329.428617] Lustre: fs13549-MDT0001: new disk, initializing [11329.434926] Lustre: Skipped 3 previous similar messages [11329.605666] Lustre: srv-fs13549-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11329.620683] Lustre: 294409:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13549-MDT0001-mdtlov: can't create new llog: rc = -115 [11329.635334] LustreError: 294409:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13549-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11334.224556] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11341.402219] Lustre: server umount fs13549-MDT0001 complete [11354.994734] loop1: detected capacity change from 0 to 204800000 [11355.645014] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11360.677673] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.748544] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11361.165252] Lustre: 295976:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13549-MDT0001-mdtlov: can't create new llog: rc = -115 [11361.176482] LustreError: 295976:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13549-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11365.742984] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11371.156413] Lustre: server umount fs13549-MDT0001 complete [11381.221021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11381.226714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11381.237649] Lustre: Skipped 4 previous similar messages [11381.258752] Lustre: Skipped 2 previous similar messages [11387.095990] Lustre: server umount lustre-MDT0000 complete [11396.958816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11397.183987] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11402.751164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11403.994649] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11413.875076] loop1: detected capacity change from 0 to 204800000 [11420.578137] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11421.006130] Lustre: 298423:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13549-MDT0001-mdtlov: can't create new llog: rc = -115 [11421.019600] LustreError: 298423:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13549-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11425.537453] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11431.193623] Lustre: server umount fs13549-MDT0001 complete [11459.040988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11459.055635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11459.055644] Lustre: Skipped 4 previous similar messages [11462.967397] Lustre: server umount lustre-OST0000 complete [11473.656359] Lustre: server umount lustre-MDT0000 complete [11477.228831] LustreError: 299814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391717 with bad export cookie 8603138198434003692 [11477.237152] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11477.240372] LustreError: 299814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11477.606918] Lustre: server umount lustre-MDT0001 complete [11490.417918] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [11493.231895] Key type lgssc unregistered [11493.546942] LNet: 300770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11493.558098] LNetError: 300770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11493.571975] LNet: Removed LNI 192.168.204.151@tcp [11494.568241] Key type .llcrypt unregistered [11494.577277] Key type ._llcrypt unregistered [11516.042378] Key type ._llcrypt registered [11516.044667] Key type .llcrypt registered [11516.139597] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [11517.403028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11517.475028] alg: No test for adler32 (adler32-zlib) [11518.745721] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [11519.057974] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [11520.711288] Key type lgssc registered [11522.192305] Lustre: Echo OBD driver; http://www.lustre.org/ [11532.385799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11532.412200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11533.948622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11539.277601] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11543.527248] LustreError: 302045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.549574] LustreError: 302045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11548.507451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11548.642534] LustreError: 302046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11548.984979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11553.966234] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11562.108785] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11568.226085] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11576.574788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11577.062919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11579.117115] LustreError: 303873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11584.484096] LustreError: 303974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11584.559658] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11589.616196] LustreError: 303873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.366655] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11598.831245] LustreError: 303873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.861952] LustreError: 303873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11613.760222] loop1: detected capacity change from 0 to 204800000 [11614.028712] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11614.041802] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11614.074085] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11616.641686] LustreError: 303974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11616.669065] LustreError: 303974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11620.437393] loop1: detected capacity change from 0 to 204800000 [11627.432275] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11627.606878] Lustre: fs24712-OST0000: new disk, initializing [11627.612997] Lustre: srv-fs24712-OST0000: No data found on store. Initialize space. [11627.621431] Lustre: 305783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11627.724680] Lustre: fs24712-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11635.879983] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11645.175562] Lustre: server umount fs24712-OST0000 complete [11650.031119] LustreError: 304551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11650.057095] LustreError: 304551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [11659.232755] loop1: detected capacity change from 0 to 204800000 [11660.266196] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11665.618467] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11665.830379] Lustre: fs24712-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11674.136890] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11675.648039] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11685.384601] loop2: detected capacity change from 0 to 204800000 [11685.951090] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11685.968981] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11685.981962] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11693.089470] loop2: detected capacity change from 0 to 204800000 [11700.249916] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11700.406163] LustreError: fs24712-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11700.429224] LustreError: fs24712-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11700.438824] LustreError: 308824:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11700.443884] LustreError: 308824:0:(tgt_mount.c:2129:server_put_super()) no obd fs24712-OST0001 [11700.450109] LustreError: 308824:0:(tgt_mount.c:116:server_deregister_mount()) fs24712-OST0001 not registered [11700.485368] Lustre: server umount fs24712-OST0001 complete [11700.487634] LustreError: 308824:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11712.826103] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11712.939683] Lustre: fs24712-OST0001: new disk, initializing [11712.945532] Lustre: srv-fs24712-OST0001: No data found on store. Initialize space. [11712.954732] Lustre: 309377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11713.056644] Lustre: fs24712-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11716.582569] LustreError: 303874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11716.600579] LustreError: 303874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11720.009963] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11729.397462] Lustre: server umount fs24712-OST0000 complete [11740.027784] Lustre: server umount fs24712-OST0001 complete [11753.609100] loop1: detected capacity change from 0 to 204800000 [11760.236461] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11760.597333] Lustre: fs24712-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11767.180037] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11775.064083] loop2: detected capacity change from 0 to 204800000 [11782.518969] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11790.536688] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11801.955371] Lustre: server umount fs24712-OST0000 complete [11811.115818] Lustre: server umount fs24712-OST0001 complete [11813.868783] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11827.174415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11827.187273] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11827.187575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11827.194545] Lustre: Skipped 1 previous similar message [11831.659361] Lustre: server umount lustre-OST0000 complete [11835.873939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11835.878585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.903778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11835.910512] Lustre: Skipped 1 previous similar message [11837.411357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11837.414286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11837.435746] Lustre: Skipped 1 previous similar message [11841.597519] Lustre: server umount lustre-MDT0000 complete [11844.844084] LustreError: 302025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392085 with bad export cookie 7444048917741460322 [11844.851813] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11844.860460] LustreError: 302025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11845.157984] Lustre: server umount lustre-MDT0001 complete [11855.479925] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [11858.371222] Key type lgssc unregistered [11858.689815] LNet: 315303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11858.703279] LNetError: 315303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11858.712921] LNet: Removed LNI 192.168.204.151@tcp [11859.691212] Key type .llcrypt unregistered [11859.694768] Key type ._llcrypt unregistered [11891.473358] Key type ._llcrypt registered [11891.475636] Key type .llcrypt registered [11891.565317] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [11908.412945] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [11909.585792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11909.672183] alg: No test for adler32 (adler32-zlib) [11910.771720] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [11911.027773] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [11912.744066] Key type lgssc registered [11913.970192] Lustre: Echo OBD driver; http://www.lustre.org/ [11955.261277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11955.275464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11956.472874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11956.505035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11956.600913] Lustre: lustre-MDT0000: new disk, initializing [11956.658624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11956.674463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11961.132572] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11972.536276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.674335] Lustre: 319166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11972.724364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11972.728269] Lustre: Skipped 1 previous similar message [11972.786668] Lustre: lustre-MDT0001: new disk, initializing [11972.875988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11972.932306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11972.952498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11977.784625] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [11987.395706] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11993.123623] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12001.024499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12001.284087] Lustre: lustre-OST0000: new disk, initializing [12001.287624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12001.295279] Lustre: 321134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12001.412509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12007.634385] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12009.021819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12009.036515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12009.155277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12017.630988] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12019.787364] Lustre: Mounted lustre-client [12020.646339] Lustre: Unmounted lustre-client [12029.923371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12029.933687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12029.947865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12029.962795] Lustre: Skipped 1 previous similar message [12035.041096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12035.054993] Lustre: Skipped 1 previous similar message [12040.165457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12040.173044] Lustre: Skipped 1 previous similar message [12044.255140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12044.577954] Lustre: server umount lustre-OST0000 complete [12049.888561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12049.904158] LustreError: Skipped 1 previous similar message [12049.908479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.916057] Lustre: Skipped 1 previous similar message [12049.920749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12054.874339] Lustre: server umount lustre-MDT0000 complete [12058.762430] LustreError: 319158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392299 with bad export cookie 10555488838041558368 [12058.765321] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12058.776871] LustreError: 319158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12059.217476] Lustre: server umount lustre-MDT0001 complete [12069.526844] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12072.288737] Key type lgssc unregistered [12081.056070] LNet: 1 local NIs in recovery (showing 1): 192.168.204.151@tcp [12081.824451] LNet: 323294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12081.844565] LNetError: 323294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12150.502262] LNet: Removed LNI 192.168.204.151@tcp [12151.399450] Key type .llcrypt unregistered [12151.401992] Key type ._llcrypt unregistered [12169.515553] Key type ._llcrypt registered [12169.518434] Key type .llcrypt registered [12169.613253] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [12188.370092] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [12189.869326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12189.980926] alg: No test for adler32 (adler32-zlib) [12191.024964] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [12191.304730] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [12192.983322] Key type lgssc registered [12194.283481] Lustre: Echo OBD driver; http://www.lustre.org/ [12230.165722] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 05:54:28 (1787392468) [12239.007813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12239.059413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.611431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12240.697967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12240.916916] Lustre: lustre-MDT0000: new disk, initializing [12241.112133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12241.133446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12246.449702] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12258.491713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.637644] Lustre: 326519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12258.722640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12258.727026] Lustre: Skipped 1 previous similar message [12258.835615] Lustre: lustre-MDT0001: new disk, initializing [12258.956137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12259.017176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12259.031510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12264.599118] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12275.094574] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12281.593995] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12289.526953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12289.768454] Lustre: lustre-OST0000: new disk, initializing [12289.775422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12289.790856] Lustre: 328490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12289.881061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12296.285247] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12298.811968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12298.822716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12299.010081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12307.212625] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12324.832745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12324.841652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.856394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12324.861510] Lustre: Skipped 1 previous similar message [12329.804290] Lustre: server umount lustre-OST0000 complete [12335.075770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.089839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12335.090944] Lustre: Skipped 2 previous similar messages [12335.104511] Lustre: Skipped 1 previous similar message [12340.196484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.211027] Lustre: Skipped 1 previous similar message [12344.292473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12344.300312] Lustre: Skipped 1 previous similar message [12348.383549] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12348.586686] Lustre: server umount lustre-MDT0000 complete [12352.244962] LustreError: 326511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392592 with bad export cookie 9716873780353740298 [12352.245765] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12352.262523] LustreError: 326511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12352.646683] Lustre: server umount lustre-MDT0001 complete [12363.231115] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12366.489530] Key type lgssc unregistered [12366.766758] LNet: 330397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12366.790485] LNetError: 330397:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12366.816632] LNet: Removed LNI 192.168.204.151@tcp [12367.522300] Key type .llcrypt unregistered [12367.524479] Key type ._llcrypt unregistered [12387.541971] Key type ._llcrypt registered [12387.545713] Key type .llcrypt registered [12387.628406] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [12388.338522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12388.467774] alg: No test for adler32 (adler32-zlib) [12389.536535] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [12389.783729] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [12391.463170] Key type lgssc registered [12392.584550] Lustre: Echo OBD driver; http://www.lustre.org/ [12399.357194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12399.375180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12400.733056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12404.629172] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12410.855929] LustreError: 331672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12410.871782] LustreError: 331672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12412.333984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12412.684836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12417.027580] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12424.987581] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12430.665622] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12438.918600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12439.199178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12445.677408] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12453.628261] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12467.725385] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12469.488861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12474.336447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12474.353761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12474.378738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12476.321842] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [12478.434687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12478.437108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12478.475626] Lustre: Skipped 2 previous similar messages [12480.332249] Lustre: server umount lustre-MDT0000 complete [12481.406926] LustreError: 331669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12483.560725] LustreError: 331669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12483.600435] LustreError: 331669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12484.573301] LustreError: 331655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392725 with bad export cookie 2699335982792032743 [12484.579747] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12484.607509] LustreError: 331655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12486.519577] LustreError: 331667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12486.551309] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [12488.672845] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12488.694553] Lustre: Skipped 1 previous similar message [12490.934365] Lustre: server umount lustre-MDT0001 complete [12494.303232] Lustre: server umount lustre-OST0000 complete [12501.694862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12505.750192] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12508.165423] Lustre: 335851:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.151@tcp [12510.003782] Lustre: server umount MGS complete [12517.231843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12517.721828] LustreError: 336423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.741659] LustreError: 336423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12517.811237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12521.996673] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12526.341615] LustreError: 336418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12526.363243] LustreError: 336418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12529.684689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12529.958247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12533.334922] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12540.775386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12541.161546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12547.222511] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12554.273300] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12555.904291] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12561.728858] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12563.336935] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12569.063888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12569.071653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12569.073270] LustreError: Skipped 1 previous similar message [12569.092107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12569.097045] Lustre: Skipped 2 previous similar messages [12569.769160] Lustre: server umount lustre-OST0000 complete [12575.203205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12575.218148] Lustre: Skipped 1 previous similar message [12575.762492] Lustre: server umount lustre-MDT0000 complete [12579.298210] LustreError: 336403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392820 with bad export cookie 2699335982792033583 [12579.311412] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12589.630673] Lustre: server umount lustre-MDT0001 complete [12602.278976] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12605.557552] Key type lgssc unregistered [12605.897803] LNet: 339859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12605.913190] LNetError: 339859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12605.957765] LNet: Removed LNI 192.168.204.151@tcp [12606.862775] Key type .llcrypt unregistered [12606.867881] Key type ._llcrypt unregistered [12623.937878] Key type ._llcrypt registered [12623.941207] Key type .llcrypt registered [12624.045827] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [12638.870251] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [12640.085025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12640.234427] alg: No test for adler32 (adler32-zlib) [12641.244702] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [12641.466746] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [12643.119251] Key type lgssc registered [12644.254777] Lustre: Echo OBD driver; http://www.lustre.org/ [12679.688257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12679.708169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12680.965414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12681.039582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12681.152734] Lustre: lustre-MDT0000: new disk, initializing [12681.232676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12681.252771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12685.872748] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12695.953551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12696.062919] Lustre: 343033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12696.135619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12696.153790] Lustre: Skipped 1 previous similar message [12696.265625] Lustre: lustre-MDT0001: new disk, initializing [12696.354565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12696.382755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12696.396661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12701.009743] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12709.796099] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12715.564113] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12725.075468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12725.524686] Lustre: lustre-OST0000: new disk, initializing [12725.531679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12725.543354] Lustre: 345002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12725.606620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12733.362882] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12733.494112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12733.502570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12733.574130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12743.002042] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12749.090525] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12749.310363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12755.788248] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12756.043996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12759.008364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12759.014909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12759.022324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12759.061895] Lustre: Skipped 1 previous similar message [12764.130083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12764.135825] Lustre: Skipped 2 previous similar messages [12764.765212] Lustre: server umount lustre-OST0000 complete [12769.249932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12769.263777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12769.264066] Lustre: Skipped 1 previous similar message [12769.281924] Lustre: Skipped 1 previous similar message [12774.378327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12774.384645] Lustre: Skipped 1 previous similar message [12775.400408] Lustre: server umount lustre-MDT0000 complete [12779.382356] LustreError: 343830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393020 with bad export cookie 13930018046492300122 [12779.396191] LustreError: 343830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12779.398853] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12779.911992] Lustre: server umount lustre-MDT0001 complete [12789.023258] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:03:47 (1787393027) [12807.562696] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [12819.349200] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [12859.724719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12859.892851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12859.923070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12860.042416] Lustre: lustre-MDT0000: new disk, initializing [12860.117659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12860.132783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12864.808689] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12875.968242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12876.065566] Lustre: 349762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12876.109198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12876.112690] Lustre: Skipped 1 previous similar message [12876.163305] Lustre: lustre-MDT0001: new disk, initializing [12876.236763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12876.269267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12876.281892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12880.433747] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12890.937193] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12898.600533] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12904.748974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12904.918561] Lustre: lustre-OST0000: new disk, initializing [12904.922147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12904.926340] Lustre: 351729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12904.979976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12911.314249] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [12915.264040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12915.281704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12915.415085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12921.210322] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12956.646078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12956.648499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12956.652486] LustreError: Skipped 1 previous similar message [12956.678245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12956.690138] Lustre: Skipped 1 previous similar message [12960.424017] Lustre: server umount lustre-OST0000 complete [12966.880506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12966.889307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12966.889853] Lustre: Skipped 2 previous similar messages [12966.901668] Lustre: Skipped 1 previous similar message [12970.380425] Lustre: server umount lustre-MDT0000 complete [12974.445324] LustreError: 351728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393215 with bad export cookie 13930018046492300997 [12974.452202] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12974.460956] LustreError: 351728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12974.928843] Lustre: server umount lustre-MDT0001 complete [12986.089548] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [12988.959952] Key type lgssc unregistered [12989.209602] LNet: 353884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12989.216256] LNetError: 353884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12989.225565] LNet: Removed LNI 192.168.204.151@tcp [12989.993732] Key type .llcrypt unregistered [12990.007749] Key type ._llcrypt unregistered [13080.702854] Key type ._llcrypt registered [13080.704877] Key type .llcrypt registered [13080.789625] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13081.025479] Key type .llcrypt unregistered [13081.030369] Key type ._llcrypt unregistered [13162.304503] Key type ._llcrypt registered [13162.308866] Key type .llcrypt registered [13162.434704] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13162.859563] Key type .llcrypt unregistered [13162.863155] Key type ._llcrypt unregistered [13195.234698] Key type ._llcrypt registered [13195.236725] Key type .llcrypt registered [13195.362390] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13195.663234] Key type .llcrypt unregistered [13195.665070] Key type ._llcrypt unregistered [13231.500516] Key type ._llcrypt registered [13231.505320] Key type .llcrypt registered [13231.652772] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13232.140309] Key type .llcrypt unregistered [13232.148126] Key type ._llcrypt unregistered [13313.327662] Key type ._llcrypt registered [13313.335696] Key type .llcrypt registered [13313.494935] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13313.933395] Key type .llcrypt unregistered [13313.942892] Key type ._llcrypt unregistered [13347.685172] Key type ._llcrypt registered [13347.688206] Key type .llcrypt registered [13347.808577] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13348.147568] Key type .llcrypt unregistered [13348.150842] Key type ._llcrypt unregistered [13382.758868] Key type ._llcrypt registered [13382.760584] Key type .llcrypt registered [13382.910973] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13383.351059] Key type .llcrypt unregistered [13383.353218] Key type ._llcrypt unregistered [13416.940230] Key type ._llcrypt registered [13416.942346] Key type .llcrypt registered [13417.025282] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13417.286918] Key type .llcrypt unregistered [13417.295553] Key type ._llcrypt unregistered [13452.193882] Key type ._llcrypt registered [13452.198918] Key type .llcrypt registered [13452.333802] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13452.664413] Key type .llcrypt unregistered [13452.671770] Key type ._llcrypt unregistered [13485.629606] Key type ._llcrypt registered [13485.635182] Key type .llcrypt registered [13485.809647] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13486.076323] Key type .llcrypt unregistered [13486.081286] Key type ._llcrypt unregistered [13550.071733] Key type ._llcrypt registered [13550.074829] Key type .llcrypt registered [13550.200859] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13550.477377] Key type .llcrypt unregistered [13550.481424] Key type ._llcrypt unregistered [13584.245128] Key type ._llcrypt registered [13584.247308] Key type .llcrypt registered [13584.330574] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13584.602558] Key type .llcrypt unregistered [13584.610192] Key type ._llcrypt unregistered [13615.301613] Key type ._llcrypt registered [13615.305546] Key type .llcrypt registered [13615.550186] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [13616.005531] Key type .llcrypt unregistered [13616.007395] Key type ._llcrypt unregistered [13671.433147] Key type ._llcrypt registered [13671.436959] Key type .llcrypt registered [13671.549476] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [13688.719021] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [13689.774477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13689.820495] alg: No test for adler32 (adler32-zlib) [13691.007669] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [13691.216645] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [13692.871329] Key type lgssc registered [13694.087755] Lustre: Echo OBD driver; http://www.lustre.org/ [13729.994530] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13730.015374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13731.254825] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13731.277210] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13731.340237] Lustre: tst250fs-MDT0000: new disk, initializing [13731.441806] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13731.482739] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13736.372142] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [13746.066106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13746.177726] Lustre: 374199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13746.220814] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13746.223945] Lustre: Skipped 1 previous similar message [13746.288353] Lustre: tst250fs-MDT0001: new disk, initializing [13746.364794] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13746.393117] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13746.408398] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13751.173534] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [13755.717750] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13762.519442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13762.870277] Lustre: tst250fs-OST0000: new disk, initializing [13762.874754] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13762.881384] Lustre: 375831:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13763.058704] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13770.480832] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [13772.360574] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13772.384651] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13772.552716] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13783.516052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13783.792915] Lustre: tst250fs-OST0001: new disk, initializing [13783.802043] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13783.813291] Lustre: 376705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13783.948735] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13789.805371] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13789.817905] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13789.977277] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13791.370680] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [13796.590653] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13799.219556] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13799.226345] Lustre: Skipped 2 previous similar messages [13818.336696] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13818.344087] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13818.364082] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13820.898305] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13820.911641] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13820.939955] Lustre: Skipped 1 previous similar message [13826.021582] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13826.033694] Lustre: Skipped 6 previous similar messages [13831.135222] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13831.144551] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13831.149756] Lustre: Skipped 3 previous similar messages [13831.425199] Lustre: server umount tst250fs-MDT0000 complete [13835.060268] LustreError: 374191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394075 with bad export cookie 17597340637180200229 [13835.061329] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13835.067587] LustreError: 374191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13835.458196] Lustre: server umount tst250fs-MDT0001 complete [13849.487413] Lustre: server umount tst250fs-OST0000 complete [13851.426367] Lustre: 372425:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394076/real 1787394076] req@ffffa064cf606300 x1874218378663168/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13851.466632] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13851.486547] Lustre: Skipped 1 previous similar message [13853.681527] Lustre: server umount tst250fs-OST0001 complete [13873.910961] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [13881.418535] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [13915.321925] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 06:22:33 (1787394153) [13917.367240] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13936.712637] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [13948.426381] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [13982.628613] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13728 sec ========= 06:23:40 (1787394220) [13984.304669] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:23:42 (1787394222) === [13987.863866] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:23:46 (1787394226) === [14032.185636] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [14035.044501] Key type lgssc unregistered [14035.367551] LNet: 385857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14035.378802] LNetError: 385857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14036.453813] LNet: Removed LNI 192.168.204.151@tcp [14037.287338] Key type .llcrypt unregistered [14037.291655] Key type ._llcrypt unregistered