[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490688661 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003243] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008617] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011124] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014007] *** VALIDATE selinux *** [ 0.022760] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026000] *** VALIDATE tmpfs *** [ 0.026415] *** VALIDATE proc *** [ 0.028142] *** VALIDATE cgroup *** [ 0.029010] *** VALIDATE cgroup2 *** [ 0.030247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032089] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034030] Spectre V2 : User space: Vulnerable [ 0.035010] Speculative Store Bypass: Vulnerable [ 0.037522] debug: unmapping init [mem 0xffffffffaa859000-0xffffffffaa860fff] [ 0.039182] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040591] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041027] ... version: 2 [ 0.042015] ... bit width: 48 [ 0.043026] ... generic registers: 4 [ 0.044013] ... value mask: 0000ffffffffffff [ 0.045017] ... max period: 00007fffffffffff [ 0.046019] ... fixed-purpose events: 3 [ 0.047012] ... event mask: 000000070000000f [ 0.048294] rcu: Hierarchical SRCU implementation. [ 0.050559] smp: Bringing up secondary CPUs ... [ 0.051594] x86: Booting SMP configuration: [ 0.052030] .... node #0, CPUs: #1 #2 #3 [ 0.055237] smp: Brought up 1 node, 4 CPUs [ 0.057014] smpboot: Max logical packages: 1 [ 0.058018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.250032] node 0 deferred pages initialised in 188ms [ 0.255011] devtmpfs: initialized [ 0.256247] x86/mm: Memory block size: 128MB [ 0.261176] gcov: version magic: 0x41383552 [ 0.264608] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.268121] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.271469] pinctrl core: initialized pinctrl subsystem [ 0.273290] [ 0.273920] ************************************************************* [ 0.276024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279023] ** ** [ 0.281018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.283018] ** ** [ 0.285020] ** This means that this kernel is built to expose internal ** [ 0.288020] ** IOMMU data structures, which may compromise security on ** [ 0.290016] ** your system. ** [ 0.292024] ** ** [ 0.294021] ** If you see this message and you are not debugging the ** [ 0.296020] ** kernel, report this immediately to your vendor! ** [ 0.299017] ** ** [ 0.301018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.303023] ************************************************************* [ 0.305835] NET: Registered protocol family 16 [ 0.308532] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.311106] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.314093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.317184] cpuidle: using governor menu [ 0.318752] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.321673] PCI: Using configuration type 1 for base access [ 0.324131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.333069] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.335051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.339036] cryptd: max_cpu_qlen set to 1000 [ 0.342293] ACPI: Added _OSI(Module Device) [ 0.344019] ACPI: Added _OSI(Processor Device) [ 0.346023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.347019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.351740] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.358534] ACPI: Interpreter enabled [ 0.359078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.361017] ACPI: Using IOAPIC for interrupt routing [ 0.362137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.365446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.376208] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.378068] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.381031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.384107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.388485] acpiphp: Slot [2] registered [ 0.389000] acpiphp: Slot [5] registered [ 0.392246] acpiphp: Slot [6] registered [ 0.394138] acpiphp: Slot [7] registered [ 0.395105] acpiphp: Slot [8] registered [ 0.397140] acpiphp: Slot [9] registered [ 0.398299] acpiphp: Slot [10] registered [ 0.400147] acpiphp: Slot [3] registered [ 0.402144] acpiphp: Slot [4] registered [ 0.403091] acpiphp: Slot [11] registered [ 0.405110] acpiphp: Slot [12] registered [ 0.406101] acpiphp: Slot [13] registered [ 0.408124] acpiphp: Slot [14] registered [ 0.410110] acpiphp: Slot [15] registered [ 0.411074] acpiphp: Slot [16] registered [ 0.412110] acpiphp: Slot [17] registered [ 0.413084] acpiphp: Slot [18] registered [ 0.415096] acpiphp: Slot [19] registered [ 0.416100] acpiphp: Slot [20] registered [ 0.418097] acpiphp: Slot [21] registered [ 0.418992] acpiphp: Slot [22] registered [ 0.420141] acpiphp: Slot [23] registered [ 0.422116] acpiphp: Slot [24] registered [ 0.423113] acpiphp: Slot [25] registered [ 0.425114] acpiphp: Slot [26] registered [ 0.427085] acpiphp: Slot [27] registered [ 0.428124] acpiphp: Slot [28] registered [ 0.429133] acpiphp: Slot [29] registered [ 0.431106] acpiphp: Slot [30] registered [ 0.432089] acpiphp: Slot [31] registered [ 0.434070] PCI host bridge to bus 0000:00 [ 0.436019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.438024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.440019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.442035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.445026] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.447018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.448191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.450989] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.454606] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.464019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.467545] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.469020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.471017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.473015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.478709] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.482071] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.485045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.487787] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.493017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.507017] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.513015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.519309] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.526019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.531020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.546017] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.555248] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.562031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.568018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.583024] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.592000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.596016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.602017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.613018] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.622826] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.629019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.635022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.649017] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.659247] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.665016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.670020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.683000] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.689000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.696016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.702020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.717021] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.729561] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.732481] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.735427] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.738383] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.740220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.746253] iommu: Default domain type: Passthrough [ 0.747444] SCSI subsystem initialized [ 0.749159] ACPI: bus type USB registered [ 0.751124] usbcore: registered new interface driver usbfs [ 0.752072] usbcore: registered new interface driver hub [ 0.754071] usbcore: registered new device driver usb [ 0.755166] pps_core: LinuxPPS API ver. 1 registered [ 0.757013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.760074] PTP clock support registered [ 0.761119] EDAC MC: Ver: 3.0.0 [ 0.763126] PCI: Using ACPI for IRQ routing [ 0.764000] NetLabel: Initializing [ 0.765012] NetLabel: domain hash size = 128 [ 0.766009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.768083] NetLabel: unlabeled traffic allowed by default [ 0.770157] vgaarb: loaded [ 0.772258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.774017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.781817] clocksource: Switched to clocksource kvm-clock [ 0.888636] VFS: Disk quotas dquot_6.6.0 [ 0.889986] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.892702] *** VALIDATE ramfs *** [ 0.893885] *** VALIDATE hugetlbfs *** [ 0.895540] pnp: PnP ACPI init [ 0.897883] pnp: PnP ACPI: found 6 devices [ 0.914409] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917782] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.919978] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.922256] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.924934] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.927291] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.930158] NET: Registered protocol family 2 [ 0.932458] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.936877] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.941678] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.946966] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950698] TCP: Hash tables configured (established 65536 bind 65536) [ 0.953836] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.956869] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959692] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963212] NET: Registered protocol family 1 [ 0.965715] RPC: Registered named UNIX socket transport module. [ 0.967840] RPC: Registered udp transport module. [ 0.969437] RPC: Registered tcp transport module. [ 0.971062] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.973228] NET: Registered protocol family 44 [ 0.975131] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.977308] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.979488] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.981884] PCI: CLS 0 bytes, default 64 [ 0.984451] Unpacking initramfs... [ 2.405951] debug: unmapping init [mem 0xffff9d42bcc54000-0xffff9d42bffbffff] [ 2.408975] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.411318] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.414096] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.924530] Initialise system trusted keyrings [ 2.926431] Key type blacklist registered [ 2.928561] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.939235] zbud: loaded [ 2.942143] *** VALIDATE nfs *** [ 2.943432] *** VALIDATE nfs4 *** [ 2.945178] pstore: using deflate compression [ 2.948610] Platform Keyring initialized [ 3.062291] NET: Registered protocol family 38 [ 3.065143] Key type asymmetric registered [ 3.066949] Asymmetric key parser 'x509' registered [ 3.068843] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.071373] io scheduler mq-deadline registered [ 3.073292] io scheduler kyber registered [ 3.075249] io scheduler bfq registered [ 3.077740] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.081099] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.084353] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.087923] ACPI: Power Button [PWRF] [ 3.175792] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.266781] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.459980] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.554965] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.747200] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.776514] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.807504] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.818338] Non-volatile memory driver v1.3 [ 3.819972] Linux agpgart interface v0.103 [ 3.856630] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.859391] vda: detected capacity change from 0 to 68526080 [ 3.878500] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.881193] vdb: detected capacity change from 0 to 1073741824 [ 3.898821] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.903101] vdc: detected capacity change from 0 to 2621440000 [ 3.920269] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.924769] vdd: detected capacity change from 0 to 2621440000 [ 3.942558] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.945829] vde: detected capacity change from 0 to 4294967296 [ 3.962219] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.965286] vdf: detected capacity change from 0 to 4294967296 [ 3.972761] libphy: Fixed MDIO Bus: probed [ 3.979821] usbcore: registered new interface driver usbserial_generic [ 3.982783] usbserial: USB Serial support registered for generic [ 3.985338] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.989286] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.990883] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.993319] mousedev: PS/2 mouse device common for all mice [ 3.996451] rtc_cmos 00:05: RTC can wake from S4 [ 3.999401] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.000952] rtc_cmos 00:05: registered as rtc0 [ 4.004581] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.007684] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.008878] intel_pstate: CPU model not supported [ 4.014161] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.019187] hid: raw HID events driver (C) Jiri Kosina [ 4.021663] usbcore: registered new interface driver usbhid [ 4.023691] usbhid: USB HID core driver [ 4.025478] drop_monitor: Initializing network drop monitor service [ 4.027918] Initializing XFRM netlink socket [ 4.030254] NET: Registered protocol family 10 [ 4.034926] Segment Routing with IPv6 [ 4.036799] NET: Registered protocol family 17 [ 4.038953] mpls_gso: MPLS GSO support [ 4.043783] RAS: Correctable Errors collector initialized. [ 4.045986] AVX version of gcm_enc/dec engaged. [ 4.047792] AES CTR mode by8 optimization enabled [ 4.142596] sched_clock: Marking stable (4142557444, 0)->(5040677262, -898119818) [ 4.146370] registered taskstats version 1 [ 4.149234] Loading compiled-in X.509 certificates [ 4.151074] zswap: loaded using pool lzo/zbud [ 4.176022] Key type big_key registered [ 4.188699] Key type encrypted registered [ 4.190315] ima: No TPM chip found, activating TPM-bypass! [ 4.192206] ima: Allocated hash algorithm: sha1 [ 4.193695] ima: No architecture policies found [ 4.195463] evm: Initialising EVM extended attributes: [ 4.197159] evm: security.selinux [ 4.198278] evm: security.ima [ 4.199308] evm: security.capability [ 4.200584] evm: HMAC attrs: 0x1 [ 4.203054] rtc_cmos 00:05: setting system clock to 2025-11-17 00:51:47 UTC (1763340707) [ 4.209617] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 4.213487] debug: unmapping init [mem 0xffffffffaa582000-0xffffffffaa858fff] [ 4.223172] Write protecting the kernel read-only data: 28672k [ 4.227026] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 4.229488] debug: unmapping init [mem 0xffffffffa9514000-0xffffffffa95fffff] [ 4.272033] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.280745] systemd[1]: Detected virtualization kvm. [ 4.282574] systemd[1]: Detected architecture x86-64. [ 4.284506] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.315547] systemd[1]: No hostname configured. [ 4.317642] systemd[1]: Set hostname to . [ 4.319813] random: systemd: uninitialized urandom read (16 bytes read) [ 4.322357] systemd[1]: Initializing machine ID from random generator. [ 4.369545] random: ln: uninitialized urandom read (6 bytes read) [ 4.467329] random: systemd: uninitialized urandom read (16 bytes read) [ 4.470078] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.475532] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.479968] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.106600] device-mapper: uevent: version 1.0.3 [ 5.108861] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.889996] random: fast init done [ 5.927986] virtio_net virtio0 ens2: renamed from eth0 [ 6.189287] scsi host0: ata_piix [ 6.219151] scsi host1: ata_piix [ 6.220627] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.222592] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.891419] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.705867] random: crng init done [ 10.707055] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.282859] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.507172] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.759545] SELinux: Disabled at runtime. [ 12.839882] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.847886] systemd[1]: Detected virtualization kvm. [ 12.849369] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.336251] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.339901] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.346729] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.351257] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.354610] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.361814] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.369372] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 13.482477] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.865694] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.218596] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.270918] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.423877] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.436525] EDAC sbridge: Ver: 1.1.2 [ 16.086657] Key type dns_resolver registered [ 16.396159] NFS: Registering the id_resolver key type [ 16.397694] Key type id_resolver registered [ 16.399055] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg141-server login: [ 41.855586] libcfs: loading out-of-tree module taints kernel. [ 41.869088] Key type ._llcrypt registered [ 41.870735] Key type .llcrypt registered [ 41.923752] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 50.287320] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 51.039606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.048084] alg: No test for adler32 (adler32-zlib) [ 52.088825] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 52.438502] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 54.071179] Key type lgssc registered [ 55.200985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.075083] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 69.276815] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.773603] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.854724] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.189786] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.900707] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 111.651329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.701822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 111.728413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 112.959779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 112.982716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 113.042603] Lustre: lustre-MDT0000: new disk, initializing [ 113.106675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.119356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 115.957784] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 126.426264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 126.483442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 126.561795] Lustre: 6489:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 126.589068] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 126.592388] Lustre: Skipped 1 previous similar message [ 126.682421] Lustre: lustre-MDT0001: new disk, initializing [ 126.759674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 126.793779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 126.801802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 129.539991] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.966732] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 141.686743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.763376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 142.008719] Lustre: lustre-OST0000: new disk, initializing [ 142.018707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 142.074620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.972507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 146.985649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 147.055590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 147.579466] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.957026] hrtimer: interrupt took 9472889 ns [ 161.656241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.711954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 161.821570] Lustre: lustre-OST0001: new disk, initializing [ 161.832678] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 161.914060] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.125276] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.038699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 172.051272] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 172.071663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 178.156676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 183.497263] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 192.622175] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing check_logdir /tmp/testlogs/ [ 195.687373] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing yml_node [ 198.631841] Lustre: DEBUG MARKER: Client: 2.16.61.48 [ 200.597398] Lustre: DEBUG MARKER: MDS: 2.16.61.48 [ 202.335952] Lustre: DEBUG MARKER: OSS: 2.16.61.48 [ 203.520819] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 19:55:05 EST 2025 [ 218.015648] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 219.311865] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 233.441185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 233.443041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 233.473680] Lustre: Skipped 1 previous similar message [ 238.561073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 238.564218] Lustre: Skipped 5 previous similar messages [ 239.603565] LustreError: 12191:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 239.696337] Lustre: server umount lustre-MDT0000 complete [ 242.358842] LustreError: 6482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340945 with bad export cookie 5413976058011340442 [ 242.361133] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 242.364353] LustreError: 6482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 243.681126] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 243.682047] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 243.688278] Lustre: Skipped 3 previous similar messages [ 243.694591] Lustre: Skipped 1 previous similar message [ 247.714557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 248.558049] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 248.572796] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 248.786698] Lustre: server umount lustre-MDT0001 complete [ 251.995202] LustreError: 12595:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 252.000812] LustreError: 12595:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 252.081864] Lustre: server umount lustre-OST0000 complete [ 255.258282] LustreError: 12795:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 255.264815] LustreError: 12795:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 255.379774] Lustre: server umount lustre-OST0001 complete [ 259.864753] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 266.109730] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 272.735991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 278.828420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 285.577943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 292.684249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 298.311374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 298.382893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 298.612539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 298.650570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 298.736497] Lustre: lustre-MDT0000: new disk, initializing [ 298.831571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 298.865878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 301.960375] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 310.982674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 311.033594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 311.103704] Lustre: 14976:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 311.162604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 311.165876] Lustre: Skipped 1 previous similar message [ 311.269466] Lustre: lustre-MDT0001: new disk, initializing [ 311.351440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 311.401882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 311.420547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 313.863603] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 319.875100] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 323.593743] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 329.037978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 329.088207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 329.312814] Lustre: lustre-OST0000: new disk, initializing [ 329.317149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 329.388228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 331.134450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 331.150224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 331.240368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 334.718654] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 341.675917] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 345.579703] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 345.785639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 349.178654] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 349.366174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 351.715556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 351.719186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.724295] LustreError: Skipped 1 previous similar message [ 351.732586] Lustre: Skipped 1 previous similar message [ 351.735310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 351.738241] Lustre: Skipped 1 previous similar message [ 357.378523] LustreError: 17918:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 357.383340] LustreError: 17918:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 357.469223] Lustre: server umount lustre-OST0000 complete [ 361.955236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.958204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.961840] Lustre: Skipped 1 previous similar message [ 361.969986] Lustre: Skipped 2 previous similar messages [ 366.877062] LustreError: 18120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 366.880898] LustreError: 18120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 366.939889] Lustre: server umount lustre-MDT0000 complete [ 367.077798] LustreError: 14989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.090445] LustreError: 14989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 369.633599] LustreError: 14969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341072 with bad export cookie 5413976058011341975 [ 369.635112] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.648612] LustreError: 14969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 369.894851] Lustre: server umount lustre-MDT0001 complete [ 374.587757] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:57:56 (1763341076) [ 389.563129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.928222] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.009915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.728290] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 395.260644] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.361563] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 402.142795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 406.046647] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 410.436613] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 413.599728] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 415.356403] Lustre: Failing over lustre-MDT0000 [ 415.500658] LustreError: 20800:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 415.508639] LustreError: 20800:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 415.602440] Lustre: server umount lustre-MDT0000 complete [ 417.760242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.771041] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 417.783817] Lustre: Skipped 1 previous similar message [ 417.795089] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 426.976981] LustreError: 20511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 426.993930] LustreError: 20511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 434.145928] Lustre: 3631:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763341121/real 1763341121] req@ffff9d420349ea00 x1848996795657600/t0(0) o400->MGC192.168.201.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763341137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 434.186274] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 434.927547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.384062] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d4202e39880 x1848996795660160/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 444.422965] LustreError: 19452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.448648] LustreError: 19452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 446.729333] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 446.743181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 446.793281] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 449.011292] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 456.250447] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 460.329181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.333402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 460.349970] Lustre: Skipped 1 previous similar message [ 460.356039] Lustre: Skipped 1 previous similar message [ 466.520464] LustreError: 21781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 466.523235] LustreError: 21781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 466.612564] Lustre: server umount lustre-MDT0000 complete [ 469.961737] LustreError: 19438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341173 with bad export cookie 5413976058011343389 [ 469.963492] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.991335] LustreError: 19438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 470.324280] Lustre: server umount lustre-MDT0001 complete [ 478.582295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.990702] LustreError: 22458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.087748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.096124] Lustre: Skipped 2 previous similar messages [ 482.934686] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 490.852740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 494.698801] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 499.642742] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 502.705471] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 504.615348] Lustre: Failing over lustre-MDT0000 [ 504.864330] Lustre: server umount lustre-MDT0000 complete [ 506.340114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.354499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.542947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.636399] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.696917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 521.706060] Lustre: Skipped 3 previous similar messages [ 524.961275] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 527.332562] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 527.334681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 527.346018] Lustre: Skipped 1 previous similar message [ 527.362749] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 531.389119] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 541.755556] LustreError: 24779:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 541.804597] LustreError: 24779:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 541.903898] Lustre: server umount lustre-MDT0000 complete [ 545.228101] LustreError: 24780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341248 with bad export cookie 5413976058011344320 [ 545.233853] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.257713] LustreError: 24780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 545.702745] Lustre: server umount lustre-MDT0001 complete [ 554.093186] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:00:55 (1763341255) [ 569.929480] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 578.312285] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 587.453093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 595.180804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 601.685857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 608.152789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 616.907740] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 627.026497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 627.080315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.368729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 627.398888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 627.468437] Lustre: lustre-MDT0000: new disk, initializing [ 627.581894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 627.586600] Lustre: Skipped 2 previous similar messages [ 627.609467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 631.100439] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 640.781638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 640.873108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 640.999493] Lustre: 28462:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 641.295620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 644.879137] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 651.618995] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 655.443232] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 663.058307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 663.138306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 665.171647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 665.213130] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 665.266945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 665.282908] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 669.633898] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 678.268243] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 683.756521] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 706.532247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 706.538522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.540390] LustreError: Skipped 1 previous similar message [ 706.541628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 706.541637] Lustre: Skipped 4 previous similar messages [ 706.567323] Lustre: Skipped 4 previous similar messages [ 708.659233] LustreError: 31126:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 708.664448] LustreError: 31126:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 708.801552] Lustre: server umount lustre-OST0000 complete [ 713.187505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.448386] Lustre: server umount lustre-MDT0000 complete [ 721.631175] LustreError: 28456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341424 with bad export cookie 5413976058011344719 [ 721.636289] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.638281] LustreError: 28456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 722.167252] Lustre: server umount lustre-MDT0001 complete [ 732.069810] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 735.370888] Key type lgssc unregistered [ 735.781812] LNet: 32061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.795947] LNetError: 32061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.836450] LNet: Removed LNI 192.168.201.141@tcp [ 736.870704] Key type .llcrypt unregistered [ 736.872516] Key type ._llcrypt unregistered [ 766.443108] Key type ._llcrypt registered [ 766.445541] Key type .llcrypt registered [ 766.567459] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 767.823297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 767.967836] alg: No test for adler32 (adler32-zlib) [ 769.001570] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 769.244903] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 770.927222] Key type lgssc registered [ 771.801982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 778.977258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 779.008488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.438605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.515200] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 789.990573] LustreError: 33458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.001561] LustreError: 33458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 790.452413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 790.671971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 794.995546] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 800.879978] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 804.879558] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 812.438421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 812.778558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.851661] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 818.093227] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 822.573587] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 825.109331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 841.450918] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:05:42 (1763341542) [ 842.961147] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 844.519932] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 850.697444] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:05:51 (1763341551) [ 860.318866] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:06:01 (1763341561) [ 861.397785] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 870.032027] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 870.046641] Lustre: Skipped 2 previous similar messages [ 871.731812] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:06:13 (1763341573) [ 875.216847] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 875.221675] Lustre: Skipped 3 previous similar messages [ 893.452943] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 893.459141] Lustre: Skipped 6 previous similar messages [ 896.495897] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:06:37 (1763341597) [ 899.471205] Lustre: *** cfs_fail_loc=131b, val=2*** [ 903.005559] Lustre: *** cfs_fail_loc=131b, val=2*** [ 911.591515] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:06:52 (1763341612) [ 913.212132] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 913.215774] Lustre: Skipped 3 previous similar messages [ 925.778705] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:07:06 (1763341626) [ 943.072157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 943.079072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.095656] LustreError: Skipped 1 previous similar message [ 943.097683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 943.146490] Lustre: Skipped 1 previous similar message [ 946.367118] LustreError: 40016:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 946.512380] Lustre: server umount lustre-OST0000 complete [ 953.313425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.316723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.321983] Lustre: Skipped 1 previous similar message [ 953.322409] Lustre: Skipped 1 previous similar message [ 956.845173] LustreError: 40217:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 956.851753] LustreError: 40217:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 956.950552] Lustre: server umount lustre-MDT0000 complete [ 958.432422] LustreError: 33458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.445255] LustreError: 33458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 960.247294] LustreError: 33439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341663 with bad export cookie 17560255346525381800 [ 960.251350] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.253663] LustreError: 33439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 960.390117] LustreError: 40419:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 960.399060] LustreError: 40419:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 960.554291] Lustre: server umount lustre-MDT0001 complete [ 968.743482] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 971.163411] Key type lgssc unregistered [ 971.396430] LNet: 40950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.404797] LNetError: 40950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.415844] LNet: Removed LNI 192.168.201.141@tcp [ 972.080732] Key type .llcrypt unregistered [ 972.082490] Key type ._llcrypt unregistered [ 988.481799] Key type ._llcrypt registered [ 988.483678] Key type .llcrypt registered [ 988.616743] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 1002.477245] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1003.443875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.476401] alg: No test for adler32 (adler32-zlib) [ 1004.536353] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1004.752065] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1006.423198] Key type lgssc registered [ 1007.405689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1015.292515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1021.425339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1027.613734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1033.207944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1038.539289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1038.577492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1038.588603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.808591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1039.840407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1039.903200] Lustre: lustre-MDT0000: new disk, initializing [ 1039.978813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.992936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1043.082575] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1051.017199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1051.072722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1051.134432] Lustre: 44079:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1051.178235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1051.183437] Lustre: Skipped 1 previous similar message [ 1051.237250] Lustre: lustre-MDT0001: new disk, initializing [ 1051.328336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1051.367889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1051.391363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1054.435204] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1060.717124] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1063.957699] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1069.723170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1069.780451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1070.135183] Lustre: lustre-OST0000: new disk, initializing [ 1070.141490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1070.245721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.075890] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1077.318723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1077.331972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1077.424722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1082.567088] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1085.626949] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1085.784691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1088.808587] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1088.984565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1093.088769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.092979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.093575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.100376] Lustre: Skipped 1 previous similar message [ 1096.506280] LustreError: 47020:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1096.583849] Lustre: server umount lustre-OST0000 complete [ 1102.817111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1102.821293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.829125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.831808] Lustre: Skipped 1 previous similar message [ 1105.190661] LustreError: 47221:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1105.193847] LustreError: 47221:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1105.243169] Lustre: server umount lustre-MDT0000 complete [ 1108.042121] LustreError: 44070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341811 with bad export cookie 17916148325179064968 [ 1108.045302] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.053887] LustreError: 44070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1108.168829] LustreError: 47422:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1108.173461] LustreError: 47422:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1108.304962] Lustre: server umount lustre-MDT0001 complete [ 1115.477329] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:10:17 (1763341817) [ 1120.251209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.636753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.500226] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1125.859769] LustreError: 48142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.872557] LustreError: 48142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1129.955973] LustreError: 48142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1130.595855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1130.908117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1134.199970] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1139.302431] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.410418] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1147.801566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1148.112937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.602823] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1156.881944] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1172.054120] LustreError: 50360:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1172.815399] Lustre: Modifying parameter general.timeout=1 in log params [ 1174.175322] Lustre: Modifying parameter general.timeout=3 in log params [ 1174.177893] Lustre: Skipped 1 previous similar message [ 1176.338504] Lustre: Modifying parameter general.timeout=6 in log params [ 1176.349102] Lustre: Skipped 2 previous similar messages [ 1180.752839] Lustre: Modifying parameter general.timeout=12 in log params [ 1180.759937] Lustre: Skipped 5 previous similar messages [ 1189.400690] Lustre: Modifying parameter general.timeout=24 in log params [ 1189.402997] Lustre: Skipped 11 previous similar messages [ 1205.602605] Lustre: Modifying parameter general.timeout=45 in log params [ 1205.605898] Lustre: Skipped 20 previous similar messages [ 1237.846042] Lustre: Modifying parameter general.timeout=89 in log params [ 1237.848807] Lustre: Skipped 43 previous similar messages [ 1272.311976] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:12:54 (1763341974) [ 1274.183235] Lustre: 57283:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1275.406068] Lustre: 57382:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1278.084300] Lustre: 57578:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1278.091182] Lustre: 57578:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1301.139790] Lustre: 59149:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1301.144248] Lustre: 59149:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1301.158287] LustreError: 59149:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1304.071731] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:13:25 (1763342005) [ 1306.875049] Lustre: Setting parameter general.jobid_var=TESTNAME in log params [ 1306.877699] Lustre: Skipped 62 previous similar messages [ 1315.296941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1315.300158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.324020] Lustre: Skipped 1 previous similar message [ 1315.328053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.332460] Lustre: Skipped 2 previous similar messages [ 1319.332749] LustreError: 59848:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1319.335305] LustreError: 59848:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1319.399106] Lustre: server umount lustre-MDT0000 complete [ 1321.953077] LustreError: 48142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.965671] LustreError: 48142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1322.434053] LustreError: 48121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342025 with bad export cookie 17916148325179065850 [ 1322.437995] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.462648] LustreError: 48121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1322.765929] Lustre: server umount lustre-MDT0001 complete [ 1334.930759] LustreError: 60250:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1334.936645] LustreError: 60250:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1335.028639] Lustre: server umount lustre-OST0000 complete [ 1342.125742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1344.936405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1347.316441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1349.840736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1354.949292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.147650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1355.354750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.701820] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1365.133343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1365.219765] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1365.245631] Lustre: 61699:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1368.054901] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1371.960897] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1375.052245] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1379.898253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1380.069576] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1384.442368] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1388.996748] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1390.859371] Lustre: 63933:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1396.195565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1396.200813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.206669] Lustre: Skipped 2 previous similar messages [ 1396.213662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.219775] Lustre: Skipped 3 previous similar messages [ 1401.329296] LustreError: 64178:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1401.337861] LustreError: 64178:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1401.413825] Lustre: server umount lustre-OST0000 complete [ 1406.432847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1406.439956] LustreError: Skipped 1 previous similar message [ 1406.442036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.453269] Lustre: Skipped 1 previous similar message [ 1406.457230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1406.463040] Lustre: Skipped 1 previous similar message [ 1410.202823] Lustre: server umount lustre-MDT0000 complete [ 1412.064564] LustreError: 63188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.072901] LustreError: 63188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1412.753263] LustreError: 61692:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342116 with bad export cookie 17916148325179068587 [ 1412.758227] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1412.763696] LustreError: 61692:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1412.999994] Lustre: server umount lustre-MDT0001 complete [ 1420.943725] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1423.122607] Key type lgssc unregistered [ 1423.341757] LNet: 65122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.352252] LNetError: 65122:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.421708] LNet: Removed LNI 192.168.201.141@tcp [ 1425.085473] Key type .llcrypt unregistered [ 1425.087064] Key type ._llcrypt unregistered [ 1445.937235] Key type ._llcrypt registered [ 1445.939165] Key type .llcrypt registered [ 1446.031803] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1446.826747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1446.870808] alg: No test for adler32 (adler32-zlib) [ 1447.880345] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1448.086923] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1449.727196] Key type lgssc registered [ 1450.528658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1456.844677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1456.867675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.280928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.277798] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1467.741495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1467.874502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1468.219314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1471.293629] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1476.339168] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1480.071977] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1485.633680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1486.004227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.718628] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1496.150741] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1512.802519] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1524.194064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1524.195289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.198500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1524.214226] Lustre: Skipped 2 previous similar messages [ 1525.595995] LustreError: 69242:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1525.652310] Lustre: server umount lustre-MDT0000 complete [ 1528.534535] LustreError: 66494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342231 with bad export cookie 7207347242389045880 [ 1528.541248] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1528.557698] LustreError: 66494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1528.715666] LustreError: 69447:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1528.722183] LustreError: 69447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1528.850703] Lustre: server umount lustre-MDT0001 complete [ 1541.756095] LustreError: 69647:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1541.760140] LustreError: 69647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1541.853786] Lustre: server umount lustre-OST0000 complete [ 1548.342418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1551.428245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1555.613337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1559.827505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1566.685126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.962444] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1566.999384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1567.228682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.794658] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1578.683084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1578.752394] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1578.771272] Lustre: 71100:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1579.235800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1582.734200] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1586.893424] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1590.064132] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1595.957603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1596.079349] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1596.482939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.749686] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1607.053673] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1609.304129] Lustre: 73342:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1611.111848] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1617.888589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1617.902069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.922672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1617.933159] Lustre: Skipped 2 previous similar messages [ 1621.835857] LustreError: 73599:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1621.864961] LustreError: 73599:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1622.023905] Lustre: server umount lustre-OST0000 complete [ 1628.640797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.644356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.655030] Lustre: Skipped 2 previous similar messages [ 1628.661662] Lustre: Skipped 3 previous similar messages [ 1632.494683] LustreError: 73800:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1632.503747] LustreError: 73800:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1632.582659] Lustre: server umount lustre-MDT0000 complete [ 1633.769112] LustreError: 71105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.784051] LustreError: 71105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1635.562654] LustreError: 71090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342338 with bad export cookie 7207347242389046699 [ 1635.568646] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1635.576358] LustreError: 71090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1635.808840] Lustre: server umount lustre-MDT0001 complete [ 1644.001963] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1645.993384] Key type lgssc unregistered [ 1646.211839] LNet: 74534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.218351] LNetError: 74534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.238500] LNet: Removed LNI 192.168.201.141@tcp [ 1646.836403] Key type .llcrypt unregistered [ 1646.838247] Key type ._llcrypt unregistered [ 1667.691519] Key type ._llcrypt registered [ 1667.696373] Key type .llcrypt registered [ 1667.792614] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1668.534336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1668.544870] alg: No test for adler32 (adler32-zlib) [ 1669.555494] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1669.734140] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1671.399207] Key type lgssc registered [ 1672.306165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1678.991333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1679.005337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.583581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.082264] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1690.087591] LustreError: 75926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.099154] LustreError: 75926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1691.335661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1691.810028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1694.982641] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1699.301251] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1702.420830] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1710.359876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1711.222702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.310886] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1723.371261] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1732.610459] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1733.369254] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1734.689392] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1734.692513] Lustre: Skipped 1 previous similar message [ 1737.000473] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1737.004622] Lustre: Skipped 2 previous similar messages [ 1741.202138] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1741.206286] Lustre: Skipped 4 previous similar messages [ 1749.619431] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1749.623180] Lustre: Skipped 10 previous similar messages [ 1765.732065] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1765.734866] Lustre: Skipped 20 previous similar messages [ 1797.927787] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1797.932287] Lustre: Skipped 45 previous similar messages [ 1813.269784] Lustre: 83693:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1815.565209] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:21:56 (1763342516) [ 1817.113702] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1818.838394] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:22:00 (1763342520) [ 1823.202294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.209180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.210065] Lustre: Skipped 1 previous similar message [ 1827.959092] LustreError: 83989:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1828.036354] Lustre: server umount lustre-OST0000 complete [ 1833.444542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.455759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.459552] Lustre: Skipped 1 previous similar message [ 1836.661021] LustreError: 84190:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1836.663282] LustreError: 84190:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1836.765631] Lustre: server umount lustre-MDT0000 complete [ 1838.562256] LustreError: 76642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.587164] LustreError: 76642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1840.002907] LustreError: 75905:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342543 with bad export cookie 16089328157408578836 [ 1840.008903] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1840.171472] LustreError: 84392:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1840.174077] LustreError: 84392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1840.291810] Lustre: server umount lustre-MDT0001 complete [ 1849.686123] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1851.976763] Key type lgssc unregistered [ 1852.201033] LNet: 84923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.205575] LNetError: 84923:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.223559] LNet: Removed LNI 192.168.201.141@tcp [ 1852.810929] Key type .llcrypt unregistered [ 1852.815336] Key type ._llcrypt unregistered [ 1856.167326] Key type ._llcrypt registered [ 1856.168736] Key type .llcrypt registered [ 1856.266283] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1859.565278] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules [ 1860.185212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1860.315607] alg: No test for adler32 (adler32-zlib) [ 1861.398047] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1861.604147] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1863.288657] Key type lgssc registered [ 1864.084544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1865.112988] LustreError: 86088:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1878.613037] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1886.442337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.962831] LustreError: 86088:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1887.000121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.638917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.832941] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1897.953143] LustreError: 87094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.965779] LustreError: 87094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1898.980353] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:23:20 (1763342600) [ 1902.787370] LustreError: 87127:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1902.792714] LustreError: 87127:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1902.797106] LustreError: 87127:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 1902.878992] LustreError: 87901:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1902.996347] Lustre: server umount lustre-MDT0000 complete [ 1913.610358] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1916.262289] Key type lgssc unregistered [ 1916.509791] LNet: 88585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.518099] LNetError: 88585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.535745] LNet: Removed LNI 192.168.201.141@tcp [ 1917.222551] Key type .llcrypt unregistered [ 1917.224192] Key type ._llcrypt unregistered [ 1931.444763] Key type ._llcrypt registered [ 1931.446639] Key type .llcrypt registered [ 1931.561923] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1932.294145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.343699] alg: No test for adler32 (adler32-zlib) [ 1933.298865] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1933.463195] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1935.111224] Key type lgssc registered [ 1935.966887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1942.848696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1942.874762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.391836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.862602] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1953.762465] LustreError: 89778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.769973] LustreError: 89778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1955.241545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1955.801868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.871850] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1963.318831] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1966.618432] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1972.880724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1973.538847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.988955] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1982.628345] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2029.183560] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:25:30 (1763342730) [ 2030.541762] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2031.908781] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:25:33 (1763342733) [ 2035.169721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.175334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.186861] Lustre: Skipped 2 previous similar messages [ 2035.189564] Lustre: Skipped 1 previous similar message [ 2040.288415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2040.301044] Lustre: Skipped 3 previous similar messages [ 2040.806266] LustreError: 92707:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2040.856368] Lustre: server umount lustre-MDT0000 complete [ 2043.835449] LustreError: 89761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342747 with bad export cookie 17280900018943679729 [ 2043.840164] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2043.847362] LustreError: 89761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2043.955813] LustreError: 92909:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2043.961503] LustreError: 92909:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2044.102309] Lustre: server umount lustre-MDT0001 complete [ 2057.614922] LustreError: 93110:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2057.625796] LustreError: 93110:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2057.875367] Lustre: server umount lustre-OST0000 complete [ 2066.899890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.236668] LustreError: 93742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.412957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.524961] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2072.546365] LustreError: 93742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.234335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2077.882674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.508382] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2086.506284] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2090.116445] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2097.275072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2104.203822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2104.271466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2104.430350] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2104.436040] LustreError: 93729:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2104.456484] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2104.467466] LustreError: 95605:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2104.481292] LustreError: 95605:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2104.486296] LustreError: 95605:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2104.574586] Lustre: server umount lustre-OST0000 complete [ 2104.579363] LustreError: 95605:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2111.925523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2112.061493] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2112.072409] LustreError: 93730:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2112.082948] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2112.090871] LustreError: 96067:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2112.095847] LustreError: 96067:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2112.101225] LustreError: 96067:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2112.155579] Lustre: server umount lustre-OST0000 complete [ 2112.158883] LustreError: 96067:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2114.447074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2119.512793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2119.708464] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2119.711256] Lustre: Found index 0 for lustre-OST0000, updating log [ 2119.718188] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2119.783198] Lustre: lustre-OST0000: new disk, initializing [ 2119.786241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2120.030752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.900567] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2128.193089] Lustre: Failing over lustre-OST0000 [ 2128.242107] LustreError: 97156:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2128.244896] LustreError: 97156:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2128.395379] Lustre: server umount lustre-OST0000 complete [ 2132.448315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.454430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.465711] Lustre: Skipped 1 previous similar message [ 2132.471306] Lustre: Skipped 1 previous similar message [ 2137.567862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.572473] Lustre: Skipped 1 previous similar message [ 2138.424646] LustreError: 97357:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2138.430880] LustreError: 97357:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2138.490256] Lustre: server umount lustre-MDT0000 complete [ 2141.282611] LustreError: 93722:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342844 with bad export cookie 17280900018943682480 [ 2141.283570] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.299156] LustreError: 93722:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2150.898246] LustreError: 97558:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2150.901569] LustreError: 97558:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2151.082864] Lustre: server umount lustre-MDT0001 complete [ 2158.982081] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:27:40 (1763342860) [ 2173.261431] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2183.216727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.710136] LustreError: 98927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.876614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.909792] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2189.282805] LustreError: 98927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.038220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2202.196735] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2205.410233] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2211.442380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2212.022195] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2216.379089] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2223.273980] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2223.314314] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2223.316313] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2224.041392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2224.107587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2224.158193] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2224.257894] Lustre: lustre-OST0001: new disk, initializing [ 2224.270865] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2224.638256] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.641441] Lustre: Skipped 2 previous similar messages [ 2224.747638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2224.754261] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2224.796489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 2230.063062] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2237.600442] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2244.648373] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2254.313364] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.332814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2254.336295] Lustre: Skipped 1 previous similar message [ 2259.420266] LustreError: 102181:0:(obd_class.h:479:obd_check_dev()) Device 26 not setup [ 2259.427817] LustreError: 102181:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2259.519704] Lustre: server umount lustre-MDT0001 complete [ 2263.264395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2263.309206] LustreError: 98922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.378266] LustreError: 98922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2270.780882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2270.876307] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2270.880187] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2270.917264] Lustre: 98915:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2272.308557] LustreError: 102761:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2272.315320] LustreError: 102761:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2272.322435] LustreError: 98919:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2272.428498] LustreError: 102772:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.141@tcp: cfg command failed: rc = -17 [ 2272.453846] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2272.453846] [ 2276.230675] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2276.862901] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2285.977555] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 20:29:47 (1763342987) [ 2301.952477] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2317.685122] Lustre: 104792:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2335.665223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2338.679843] Lustre: 105928:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2424.801142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2424.805324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.817362] Lustre: Skipped 2 previous similar messages [ 2424.820819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.824111] Lustre: Skipped 4 previous similar messages [ 2426.957893] LustreError: 106386:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2426.964881] LustreError: 106386:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2427.021203] Lustre: server umount lustre-MDT0000 complete [ 2429.636123] LustreError: 98907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343132 with bad export cookie 17280900018943683411 [ 2429.637487] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.645652] LustreError: 98907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2429.849358] Lustre: server umount lustre-MDT0001 complete [ 2441.964086] Lustre: server umount lustre-OST0000 complete [ 2455.310464] Lustre: server umount lustre-OST0001 complete [ 2460.628826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2469.611430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2470.715732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2478.577085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2485.781485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2486.624119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2505.485900] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2514.028992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2514.046277] Lustre: lustre-MDT0000: reset Object Index mappings [ 2514.403942] LustreError: 109869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.411902] LustreError: 109869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2514.509503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.512397] Lustre: Skipped 1 previous similar message [ 2517.860780] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2524.538531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2524.549733] Lustre: lustre-MDT0001: reset Object Index mappings [ 2528.072943] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2530.446378] Lustre: 110983:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2536.473480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2539.313045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2542.201094] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2547.615971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2549.258549] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:156 to 0x2c0000400:193) [ 2550.344027] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 2550.346828] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:200 to 0x2c0000401:225) [ 2551.992126] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2558.844441] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2561.957757] Lustre: 112837:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2610.526237] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:35:11 (1763343311) [ 2637.793086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2637.799529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.809615] Lustre: Skipped 3 previous similar messages [ 2637.813591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.816378] Lustre: Skipped 4 previous similar messages [ 2643.727775] LustreError: 113197:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2643.735357] LustreError: 113197:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2643.813472] Lustre: server umount lustre-MDT0000 complete [ 2647.098757] LustreError: 109849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343350 with bad export cookie 17280900018943730262 [ 2647.104495] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.108947] LustreError: 109849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2647.521685] LustreError: 109864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.524479] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.535014] LustreError: 109864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2647.540344] Lustre: Skipped 4 previous similar messages [ 2653.556875] Lustre: server umount lustre-MDT0001 complete [ 2662.781540] Lustre: server umount lustre-OST0000 complete [ 2680.287368] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2680.525077] Lustre: server umount lustre-OST0001 complete [ 2686.214918] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2694.777992] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2702.652936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.130196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2712.565330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2718.011640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2723.769589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2723.845288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.071144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2724.102280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2724.154662] Lustre: lustre-MDT0000: new disk, initializing [ 2724.227255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.234189] Lustre: Skipped 3 previous similar messages [ 2724.256673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2727.015602] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2731.978780] Lustre: server umount lustre-MDT0000 complete [ 2733.902327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2739.502902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2739.568236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.835240] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2739.870403] Lustre: 117076:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2739.889558] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2743.553760] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2747.767414] Lustre: server umount lustre-MDT0000 complete [ 2754.995680] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:37:36 (1763343456) [ 2756.232717] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2757.583570] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:37:39 (1763343459) [ 2770.610076] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2778.425247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.455247] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2789.981812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2790.059828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2790.127220] Lustre: 119095:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2790.192507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2790.195076] Lustre: Skipped 1 previous similar message [ 2790.271368] Lustre: lustre-MDT0001: new disk, initializing [ 2790.424279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2790.437710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2794.316642] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2798.456503] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2805.048501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2805.135643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2805.414209] Lustre: lustre-OST0000: new disk, initializing [ 2805.426661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2806.650840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2806.656622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2806.793706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2809.864859] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2818.514540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2818.607870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2818.772370] Lustre: lustre-OST0001: new disk, initializing [ 2818.779758] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2820.752173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2820.759508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2820.888086] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2824.384329] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2833.213441] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2835.678401] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3183.229880] Lustre: 127899:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3183.240186] Lustre: 127899:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3183.243900] Lustre: 127899:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3183.249402] Lustre: 127899:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3183.260914] Lustre: 127899:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3183.265615] Lustre: 127899:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3457.509957] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:49:18 (1763344158) [ 3471.330968] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.333181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.351672] Lustre: Skipped 2 previous similar messages [ 3471.356425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3471.377794] Lustre: Skipped 11 previous similar messages [ 3476.453374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3476.456173] Lustre: Skipped 3 previous similar messages [ 3477.215326] LustreError: 128136:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3477.218346] LustreError: 128136:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3477.359195] Lustre: server umount lustre-MDT0000 complete [ 3480.095228] LustreError: 119085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344183 with bad export cookie 17280900018943796237 [ 3480.099781] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3480.100081] LustreError: 119085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3481.571292] LustreError: 122292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.573539] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.583652] LustreError: 122292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3481.589424] Lustre: Skipped 2 previous similar messages [ 3485.604592] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3485.608278] Lustre: Skipped 3 previous similar messages [ 3486.468144] Lustre: server umount lustre-MDT0001 complete [ 3495.244107] LustreError: 128539:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3495.250468] LustreError: 128539:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3495.315554] Lustre: server umount lustre-OST0000 complete [ 3504.413027] Lustre: server umount lustre-OST0001 complete [ 3510.219458] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3517.757177] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3526.169113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.870446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3537.919574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.078105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.214489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.280131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.453243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3550.499657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3550.549403] Lustre: lustre-MDT0000: new disk, initializing [ 3550.629496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.633309] Lustre: Skipped 5 previous similar messages [ 3550.655380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3554.352842] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3564.856976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3564.921473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3564.999868] Lustre: 130924:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3565.081805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3565.086434] Lustre: Skipped 1 previous similar message [ 3565.195692] Lustre: lustre-MDT0001: new disk, initializing [ 3565.441218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3565.455672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3570.192627] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3580.499381] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3586.335969] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3594.968908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3595.070604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3595.485786] Lustre: lustre-OST0000: new disk, initializing [ 3595.498841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3596.849051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3596.860895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3596.990741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3603.253115] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3612.974350] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3623.791381] Lustre: lustre-MDD0000: changelog on [ 3626.424498] Lustre: lustre-MDD0001: changelog on [ 5600.740642] Lustre: lustre-MDD0001: changelog off [ 5602.842666] Lustre: 130932:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5603.779360] Lustre: lustre-MDD0000: changelog off [ 5612.538090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.542276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5612.581840] Lustre: Skipped 1 previous similar message [ 5617.468827] LustreError: 140771:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5617.481033] LustreError: 140771:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5617.707748] Lustre: server umount lustre-OST0000 complete [ 5622.756720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.762930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5622.784968] Lustre: Skipped 1 previous similar message [ 5622.791402] Lustre: Skipped 1 previous similar message [ 5627.872565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5627.882710] Lustre: Skipped 2 previous similar messages [ 5627.944675] LustreError: 140975:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5627.949806] LustreError: 140975:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5628.244387] Lustre: server umount lustre-MDT0000 complete [ 5631.985839] LustreError: 131773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5632.020042] LustreError: 131773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5632.319074] LustreError: 130917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346335 with bad export cookie 17280900018944607614 [ 5632.327416] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5632.334703] LustreError: 130917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5632.860476] Lustre: server umount lustre-MDT0001 complete [ 5644.841898] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5647.693455] Key type lgssc unregistered [ 5648.049593] LNet: 141709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5648.057443] LNetError: 141709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5648.078677] LNet: Removed LNI 192.168.201.141@tcp [ 5649.267171] Key type .llcrypt unregistered [ 5649.271402] Key type ._llcrypt unregistered [ 5669.565471] Key type ._llcrypt registered [ 5669.567133] Key type .llcrypt registered [ 5669.694928] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 5683.240668] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5684.378815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5684.496298] alg: No test for adler32 (adler32-zlib) [ 5685.596460] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5685.851156] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5687.521233] Key type lgssc registered [ 5688.566047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5698.294592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5705.595294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5711.667842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5717.531702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5724.839989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5724.883770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5724.895363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.099877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5726.128895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5726.208319] Lustre: lustre-MDT0000: new disk, initializing [ 5726.294755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.312294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5729.468301] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5739.724576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5739.785942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5739.849245] Lustre: 144888:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5739.901443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5739.905330] Lustre: Skipped 1 previous similar message [ 5739.958498] Lustre: lustre-MDT0001: new disk, initializing [ 5740.064091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5740.121446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5740.141747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5743.915312] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5751.115681] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5754.754994] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5760.894485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5760.968785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5761.150388] Lustre: lustre-OST0000: new disk, initializing [ 5761.155868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5761.278309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5766.193918] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5767.218250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5767.224584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5767.317571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5772.618515] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5776.765963] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_module obdecho/obdecho [ 5778.191664] LustreError: 147508:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5778.204064] LustreError: 147508:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 5779.706948] LustreError: 147604:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5791.204171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5791.216277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.240020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5792.736422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5792.738439] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.748118] Lustre: Skipped 1 previous similar message [ 5796.007769] LustreError: 147944:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5796.110262] Lustre: server umount lustre-MDT0000 complete [ 5799.770598] LustreError: 144881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346503 with bad export cookie 14667486396361112560 [ 5799.776403] LustreError: 144881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5799.776706] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5799.894453] LustreError: 148146:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5799.900388] LustreError: 148146:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5800.035837] Lustre: server umount lustre-MDT0001 complete [ 5813.583135] LustreError: 148347:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5813.585644] LustreError: 148347:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5813.758445] Lustre: server umount lustre-OST0000 complete [ 5831.561725] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 5840.522793] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5850.620228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5857.208242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5862.757561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5868.576853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5874.666757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5874.725318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5874.899844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5874.928463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5875.004963] Lustre: lustre-MDT0000: new disk, initializing [ 5875.091107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.111974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5879.626465] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5891.209705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5891.274554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5891.384098] Lustre: 151297:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5891.495961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5891.499080] Lustre: Skipped 1 previous similar message [ 5891.597777] Lustre: lustre-MDT0001: new disk, initializing [ 5891.746844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5891.819148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5891.850324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5896.103283] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5903.657452] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5906.709234] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5914.115758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5914.231519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5914.496746] Lustre: lustre-OST0000: new disk, initializing [ 5914.515934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5914.627371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5919.852272] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5923.910823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5923.929428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5924.022974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5925.904404] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5929.470689] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5929.679708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5932.956467] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5933.112661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5934.564197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5934.564367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.566335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5934.566341] Lustre: Skipped 2 previous similar messages [ 5934.568327] LustreError: Skipped 1 previous similar message [ 5934.581655] Lustre: Skipped 1 previous similar message [ 5939.684440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5939.687953] Lustre: Skipped 2 previous similar messages [ 5940.537367] LustreError: 154238:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5940.541783] LustreError: 154238:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5940.620111] Lustre: server umount lustre-OST0000 complete [ 5943.264444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5943.268216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.801724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5944.807163] Lustre: Skipped 2 previous similar messages [ 5949.232838] LustreError: 154440:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5949.238680] LustreError: 154440:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5949.921877] LustreError: 151308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.933954] LustreError: 151308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5951.302270] Lustre: server umount lustre-MDT0000 complete [ 5954.871567] LustreError: 151287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346658 with bad export cookie 14667486396361113526 [ 5954.875976] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5954.879774] LustreError: 151287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5955.302314] Lustre: server umount lustre-MDT0001 complete [ 5958.678534] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:31:00 (1763346660) [ 5959.932042] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5961.521134] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:31:02 (1763346662) [ 5966.980840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.304959] LustreError: 155315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.373935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.255253] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5972.449904] LustreError: 155314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.188437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5981.059264] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5985.701435] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5989.147828] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5994.629864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5999.292751] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6003.597701] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6023.649822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6023.657837] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.664478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.667120] Lustre: Skipped 2 previous similar messages [ 6023.672560] Lustre: Skipped 2 previous similar messages [ 6025.701883] LustreError: 157584:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6025.705111] LustreError: 157584:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6025.788817] Lustre: server umount lustre-MDT0000 complete [ 6028.574365] LustreError: 156035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346731 with bad export cookie 14667486396361114401 [ 6028.575423] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6028.602051] LustreError: 156035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6028.768353] LustreError: 155309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.891112] LustreError: 156491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.934704] LustreError: 156491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6035.118992] Lustre: server umount lustre-MDT0001 complete [ 6041.642558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6052.974741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6053.150936] LustreError: 158847:0:(llog.c:1616:llog_backup()) MGC192.168.201.141@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6053.162142] Lustre: 158847:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.141@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6058.465113] LustreError: 158855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.499199] LustreError: 158855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6075.643627] LustreError: 158854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6075.661885] LustreError: 158854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 6109.152146] LustreError: 158871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.172663] LustreError: 158871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 6113.760511] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6113.773037] Lustre: 158847:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6139.872637] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6139.881604] Lustre: 155306:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6154.731095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6154.737354] Lustre: Skipped 2 previous similar messages [ 6158.231989] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6159.851627] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6160.352333] LustreError: 155306:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6165.292303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.155588] LustreError: 158854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.160216] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcb8d612a72fa9c7a [ 6173.169288] LustreError: 158854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 45 previous similar messages [ 6173.178591] Lustre: MGC192.168.201.141@tcp: Connection restored to 0@lo (at 0@lo) [ 6177.340852] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6178.805768] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6182.039475] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6183.377722] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6186.318663] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6187.697179] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6203.593592] LustreError: 160352:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6203.596625] LustreError: 160352:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6203.649401] Lustre: server umount lustre-MDT0000 complete [ 6204.386268] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.397790] Lustre: Skipped 3 previous similar messages [ 6206.727863] LustreError: 155295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346910 with bad export cookie 14667486396361120890 [ 6206.731387] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6206.750521] LustreError: 155295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6206.971736] Lustre: server umount lustre-MDT0001 complete [ 6220.135457] Lustre: server umount lustre-OST0000 complete [ 6239.778637] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 6249.045414] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6256.850385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6262.302562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6268.279364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6273.834120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6279.434466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6279.489166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.679659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6279.706869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6279.757648] Lustre: lustre-MDT0000: new disk, initializing [ 6279.817498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.822682] Lustre: Skipped 1 previous similar message [ 6279.837626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6282.980371] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6291.605972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6291.659988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6291.721143] Lustre: 163700:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6291.944194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6294.850327] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6301.317346] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6304.564629] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6309.623816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6309.671430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6315.116340] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6317.099438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6317.104224] Lustre: Skipped 1 previous similar message [ 6317.107750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6317.221526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6322.061956] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6325.574378] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6325.753488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6329.613539] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6329.878643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6332.901830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6332.906180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6332.907486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6332.907492] Lustre: Skipped 5 previous similar messages [ 6332.921713] LustreError: Skipped 1 previous similar message [ 6332.923314] Lustre: Skipped 2 previous similar messages [ 6337.790346] LustreError: 166640:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6337.797809] LustreError: 166640:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6337.916644] Lustre: server umount lustre-OST0000 complete [ 6347.935878] Lustre: server umount lustre-MDT0000 complete [ 6348.258977] LustreError: 164798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.282782] LustreError: 164798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6351.647215] LustreError: 163693:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347054 with bad export cookie 14667486396361137704 [ 6351.650228] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6351.671326] LustreError: 163693:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6352.012175] Lustre: server umount lustre-MDT0001 complete [ 6360.825865] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:37:42 (1763347062) [ 6366.488441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.118338] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6379.237340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6383.670946] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6389.642882] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6394.452304] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6401.458391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6407.501437] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6413.321368] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6422.991850] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:38:44 (1763347124) [ 6427.620240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6427.623168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.633920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6427.633928] Lustre: Skipped 3 previous similar messages [ 6427.672203] Lustre: Skipped 3 previous similar messages [ 6432.720754] LustreError: 170155:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6432.733762] LustreError: 170155:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6432.870021] Lustre: server umount lustre-OST0000 complete [ 6442.548374] Lustre: server umount lustre-MDT0000 complete [ 6445.577687] LustreError: 167697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347148 with bad export cookie 14667486396361138586 [ 6445.580790] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6445.610929] LustreError: 167697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6445.942496] Lustre: server umount lustre-MDT0001 complete [ 6456.108545] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6458.802715] Key type lgssc unregistered [ 6459.095892] LNet: 171092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6459.105669] LNetError: 171092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6459.130151] LNet: Removed LNI 192.168.201.141@tcp [ 6460.247460] Key type .llcrypt unregistered [ 6460.249203] Key type ._llcrypt unregistered [ 6479.611423] Key type ._llcrypt registered [ 6479.614119] Key type .llcrypt registered [ 6479.708301] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6480.395575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6480.405356] alg: No test for adler32 (adler32-zlib) [ 6481.500925] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 6481.785321] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6483.471282] Key type lgssc registered [ 6484.629429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6493.773852] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6493.796679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6509.407438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6514.528266] LustreError: 172313:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 6519.647466] LustreError: 172313:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6519.670503] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6519.680729] LustreError: MGC192.168.201.141@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6519.685861] LustreError: 172313:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6519.689800] LustreError: 172313:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 6519.695544] LustreError: 172313:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6519.704513] LustreError: 172313:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6519.797476] Lustre: server umount lustre-OST0000 complete [ 6519.799312] LustreError: 172313:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6529.676286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6531.244588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.291898] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6540.779334] LustreError: 172814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.816947] LustreError: 172814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6545.889185] LustreError: 172814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.944302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6546.512065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6550.897691] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6556.605434] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6560.621931] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6567.448283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6567.637780] LustreError: 174407:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6567.643591] Lustre: 174407:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.141@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6567.814827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6572.864630] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6577.851618] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6581.217929] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.222866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6581.236983] Lustre: Skipped 1 previous similar message [ 6585.869299] LustreError: 175058:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6585.871829] LustreError: 175058:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6586.010519] Lustre: server umount lustre-MDT0000 complete [ 6586.337622] LustreError: 172815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.348938] LustreError: 172815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6589.638109] LustreError: 172796:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347292 with bad export cookie 1455802748246173296 [ 6589.657091] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6589.658719] LustreError: 172796:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6589.810516] LustreError: 175261:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6589.821968] LustreError: 175261:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6589.989675] Lustre: server umount lustre-MDT0001 complete [ 6604.619455] LustreError: 175463:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6604.625984] LustreError: 175463:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6604.750424] Lustre: server umount lustre-OST0000 complete [ 6612.528686] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 6621.603873] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6632.144327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6640.757703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6647.716318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.405355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6661.859918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6661.930752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6662.104554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6662.134613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6662.231123] Lustre: lustre-MDT0000: new disk, initializing [ 6662.401685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6662.418194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6666.759517] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6677.908850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6678.017892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6678.172719] Lustre: 177803:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6678.257761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6678.267356] Lustre: Skipped 1 previous similar message [ 6678.405428] Lustre: lustre-MDT0001: new disk, initializing [ 6678.556691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6678.606841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6678.621944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6683.083964] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6691.991706] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6695.909451] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6702.655203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6702.765183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6703.046629] Lustre: lustre-OST0000: new disk, initializing [ 6703.053536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6703.148555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.324388] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6713.406606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6713.425951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6713.544585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6717.076371] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6721.698186] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6721.898139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6726.042267] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6726.288208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6729.187338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.188891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6729.248058] Lustre: Skipped 2 previous similar messages [ 6729.262850] Lustre: Skipped 2 previous similar messages [ 6734.308261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6734.318705] Lustre: Skipped 2 previous similar messages [ 6734.424144] LustreError: 180742:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6734.426900] LustreError: 180742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6734.558088] Lustre: server umount lustre-OST0000 complete [ 6739.426652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.431134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6739.442185] Lustre: Skipped 1 previous similar message [ 6744.547533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6744.553478] Lustre: Skipped 2 previous similar messages [ 6753.247162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6753.411834] LustreError: 180946:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6753.421800] LustreError: 180946:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6753.531436] Lustre: server umount lustre-MDT0000 complete [ 6754.785097] LustreError: 180745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6754.801803] LustreError: 180745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6757.104338] LustreError: 180743:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347460 with bad export cookie 1455802748246174024 [ 6757.107760] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6757.135380] LustreError: 180743:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6757.447383] Lustre: server umount lustre-MDT0001 complete [ 6767.312456] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:44:28 (1763347468) [ 6784.536727] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6786.985596] Key type lgssc unregistered [ 6787.278725] LNet: 182332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6787.293218] LNetError: 182332:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6787.313824] LNet: Removed LNI 192.168.201.141@tcp [ 6788.153319] Key type .llcrypt unregistered [ 6788.154944] Key type ._llcrypt unregistered [ 6806.552779] Key type ._llcrypt registered [ 6806.557728] Key type .llcrypt registered [ 6806.649248] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6807.815589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6807.932199] alg: No test for adler32 (adler32-zlib) [ 6808.944952] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 6809.144796] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6810.831620] Key type lgssc registered [ 6811.864914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6818.897532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6818.914442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.410103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6823.762860] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6830.567440] LustreError: 183534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.583695] LustreError: 183534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6832.689059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6833.000478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6837.167967] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6842.656363] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6846.500981] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6854.722447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6855.012299] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6855.015181] LustreError: 185175:0:(llog.c:1657:llog_backup()) MGC192.168.201.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6855.025319] Lustre: 185175:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.141@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6855.167377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.318708] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6868.142223] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6874.084210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6874.090044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.103430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6875.623627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.628678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6875.674877] Lustre: Skipped 1 previous similar message [ 6875.696188] Lustre: Skipped 2 previous similar messages [ 6880.738318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.753984] Lustre: Skipped 2 previous similar messages [ 6885.343248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6885.472234] LustreError: 185828:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6885.565232] Lustre: server umount lustre-MDT0000 complete [ 6885.874517] LustreError: 184552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.928132] LustreError: 184552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6890.287255] LustreError: 183514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347593 with bad export cookie 2978876838748554906 [ 6890.315869] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6890.515604] LustreError: 186032:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6890.518352] LustreError: 186032:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6890.733340] Lustre: server umount lustre-MDT0001 complete [ 6905.309396] LustreError: 186233:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6905.312625] LustreError: 186233:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6905.647740] Lustre: server umount lustre-OST0000 complete [ 6918.197226] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 6930.592627] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6942.644852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6950.274584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6956.742615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6963.326402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6970.502583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6970.601202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.866327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6970.905742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6970.981555] Lustre: lustre-MDT0000: new disk, initializing [ 6971.096723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.121709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6974.975296] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6985.528033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6985.587996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6985.674837] Lustre: 188575:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6985.710543] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6985.716118] Lustre: Skipped 1 previous similar message [ 6985.718851] LustreError: 189342:0:(llog.c:1657:llog_backup()) MGC192.168.201.141@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6985.724586] LustreError: 189342:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6985.728895] Lustre: 189342:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.141@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6985.743962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6985.746881] Lustre: Skipped 1 previous similar message [ 6985.796151] Lustre: lustre-MDT0001: new disk, initializing [ 6985.887944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6985.931812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6985.944749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6989.502965] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6997.394624] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7001.968702] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7009.062165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7009.139454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7009.334192] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7009.336426] Lustre: Skipped 1 previous similar message [ 7009.339051] LustreError: 190297:0:(llog.c:1657:llog_backup()) MGC192.168.201.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7009.344815] LustreError: 190297:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 7009.349657] Lustre: 190297:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.141@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7009.362792] Lustre: 190297:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7009.380208] Lustre: lustre-OST0000: new disk, initializing [ 7009.385344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7009.477455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7016.439430] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7019.565246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7019.581354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7019.651344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7024.056903] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7028.717822] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7028.906689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7033.295741] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7033.476816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7035.360905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7035.364091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.395445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7035.405375] Lustre: Skipped 1 previous similar message [ 7039.978548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7039.984543] Lustre: Skipped 1 previous similar message [ 7041.228191] LustreError: 191514:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7041.241939] LustreError: 191514:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7041.347800] Lustre: server umount lustre-OST0000 complete [ 7047.654454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7047.658086] LustreError: Skipped 1 previous similar message [ 7047.665300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7047.678635] Lustre: Skipped 1 previous similar message [ 7050.220909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.229544] Lustre: Skipped 1 previous similar message [ 7051.523733] LustreError: 191715:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7051.536309] LustreError: 191715:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7051.626580] Lustre: server umount lustre-MDT0000 complete [ 7055.340858] LustreError: 188587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.380940] LustreError: 188587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7055.595029] LustreError: 188567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347758 with bad export cookie 2978876838748555627 [ 7055.602642] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7055.623114] LustreError: 188567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7055.966350] Lustre: server umount lustre-MDT0001 complete [ 7066.081677] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:49:27 (1763347767) [ 7083.817302] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7094.452254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.865484] LustreError: 193287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.919705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7099.513690] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7100.384262] LustreError: 193287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7109.835717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7115.277578] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7118.255760] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7125.812340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7132.817047] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7142.439954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7142.522769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7142.802360] Lustre: lustre-OST0001: new disk, initializing [ 7142.821946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7142.936633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7142.951480] Lustre: Skipped 2 previous similar messages [ 7149.754393] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7151.724255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7151.735983] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7151.902149] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7161.081556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7168.882496] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7176.293695] Lustre: DEBUG MARKER: ADD OST3 [ 7181.845459] loop: module loaded [ 7181.952281] loop0: detected capacity change from 0 to 204800000 [ 7182.204127] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7182.211170] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7182.289914] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7182.410239] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7188.606239] loop0: detected capacity change from 0 to 204800000 [ 7189.765679] Lustre: DEBUG MARKER: STOP OST3 [ 7194.933904] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7195.010236] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7195.181333] Lustre: lustre-OST0003: new disk, initializing [ 7195.187708] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7196.828333] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7197.047864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7197.063168] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7197.103448] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7198.879115] LustreError: 197711:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7199.567207] Lustre: Failing over lustre-OST0003 [ 7199.663118] LustreError: 197914:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 7199.677795] LustreError: 197914:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7199.778324] Lustre: server umount lustre-OST0003 complete [ 7202.282668] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7202.289956] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.299377] Lustre: Skipped 1 previous similar message [ 7202.302497] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7202.313876] LustreError: 195989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7202.329497] LustreError: 195989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7212.090871] Lustre: DEBUG MARKER: START OST3 again [ 7218.550450] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7218.915535] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7218.928310] Lustre: Skipped 1 previous similar message [ 7218.952417] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7220.197927] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7220.342575] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7220.354572] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7224.858924] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7247.179977] Lustre: Failing over lustre-OST0003 [ 7247.240187] LustreError: 199896:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 7247.256528] LustreError: 199896:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7247.321977] Lustre: server umount lustre-OST0003 complete [ 7249.899110] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.906834] LustreError: 194744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.914859] Lustre: Skipped 2 previous similar messages [ 7249.931414] LustreError: 194744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 7265.255028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.379505] Lustre: server umount lustre-MDT0000 complete [ 7275.420890] LustreError: 196369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347978 with bad export cookie 2978876838748556509 [ 7275.436414] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7275.451521] LustreError: 196369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7275.501850] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.519021] Lustre: Skipped 4 previous similar messages [ 7281.805643] LustreError: 200368:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 7281.833687] LustreError: 200368:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7282.101225] Lustre: server umount lustre-MDT0001 complete [ 7286.185732] Lustre: server umount lustre-OST0000 complete [ 7290.812229] Lustre: server umount lustre-OST0001 complete [ 7297.430840] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7304.914466] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7314.325054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7320.808186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7327.297367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7333.935894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7341.090495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7341.160606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.427031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7341.471110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7341.561529] Lustre: lustre-MDT0000: new disk, initializing [ 7341.695667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7346.739704] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7358.757490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7358.822274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7358.890336] Lustre: 202958:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7358.957103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7358.959991] Lustre: Skipped 1 previous similar message [ 7359.083690] Lustre: lustre-MDT0001: new disk, initializing [ 7359.240800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7359.251583] Lustre: Skipped 1 previous similar message [ 7359.342677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7359.347057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7364.176893] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7371.069804] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7374.547733] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7382.286377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7382.349793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7383.931761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7384.092618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7389.243830] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7397.879236] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7402.252842] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7402.481418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7406.043034] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7406.253811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7409.640474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7409.641495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.643562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7409.643571] Lustre: Skipped 11 previous similar messages [ 7409.649943] LustreError: Skipped 1 previous similar message [ 7409.656600] Lustre: Skipped 2 previous similar messages [ 7414.013110] LustreError: 205898:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7414.017621] LustreError: 205898:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 7414.154079] Lustre: server umount lustre-OST0000 complete [ 7424.336084] Lustre: server umount lustre-MDT0000 complete [ 7425.007329] LustreError: 202966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7425.044352] LustreError: 202966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 7428.173874] LustreError: 202950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348131 with bad export cookie 2978876838748558805 [ 7428.176587] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.201805] LustreError: 202950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7428.649989] Lustre: server umount lustre-MDT0001 complete [ 7432.387487] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:55:33 (1763348133) [ 7446.766186] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7455.750973] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7467.008898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7473.981591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7480.186814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7485.960869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7492.754765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7492.815819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7493.016684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7493.038759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7493.041941] Lustre: Skipped 1 previous similar message [ 7493.113077] Lustre: lustre-MDT0000: new disk, initializing [ 7493.115265] Lustre: Skipped 1 previous similar message [ 7493.219121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7493.224713] Lustre: Skipped 1 previous similar message [ 7496.945647] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7506.461439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7506.555339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7506.712241] Lustre: 209146:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7506.966624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7510.462360] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7517.913623] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7521.210573] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7527.787126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7527.850478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7529.988274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7533.507665] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7541.941304] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7546.224710] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7546.412417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7550.993910] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7551.254682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7555.556738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7555.558788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.567928] LustreError: Skipped 1 previous similar message [ 7555.569939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7555.569947] Lustre: Skipped 3 previous similar messages [ 7555.604823] Lustre: Skipped 3 previous similar messages [ 7567.330152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7567.411366] LustreError: 212089:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7567.417802] LustreError: 212089:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7567.548833] Lustre: server umount lustre-OST0000 complete [ 7573.472470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7578.010966] Lustre: server umount lustre-MDT0000 complete [ 7581.154957] LustreError: 209155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.168464] LustreError: 209155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7582.557562] LustreError: 209929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348285 with bad export cookie 2978876838748559687 [ 7582.568872] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7582.596613] LustreError: 209929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7582.996501] Lustre: server umount lustre-MDT0001 complete [ 7592.126209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.189229] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7604.422791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7609.498530] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7614.758814] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7618.215227] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7625.027089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7625.397500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7625.400653] Lustre: Skipped 6 previous similar messages [ 7630.872284] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7636.727037] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7648.227212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7653.507466] Lustre: server umount lustre-OST0000 complete [ 7663.931864] Lustre: server umount lustre-MDT0000 complete [ 7667.702367] LustreError: 212950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348370 with bad export cookie 2978876838748560569 [ 7667.722205] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7667.725453] LustreError: 212950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7668.125452] Lustre: server umount lustre-MDT0001 complete [ 7777.040679] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7779.845211] Key type lgssc unregistered [ 7780.156558] LNet: 216291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7780.173247] LNetError: 216291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7780.189358] LNet: Removed LNI 192.168.201.141@tcp [ 7781.085164] Key type .llcrypt unregistered [ 7781.087950] Key type ._llcrypt unregistered [ 7803.164167] Key type ._llcrypt registered [ 7803.166948] Key type .llcrypt registered [ 7803.260502] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7817.623451] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7819.210991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7819.220066] alg: No test for adler32 (adler32-zlib) [ 7820.352371] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 7820.567655] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7822.207226] Key type lgssc registered [ 7823.360493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7832.455536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7839.225153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7845.111549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7851.900723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7859.946975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7859.997808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7860.011861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.200149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7861.260137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7861.376712] Lustre: lustre-MDT0000: new disk, initializing [ 7861.461120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7861.491735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7865.323738] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7875.637236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7875.760263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7875.880650] Lustre: 219624:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7875.931516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7875.934873] Lustre: Skipped 1 previous similar message [ 7876.001909] Lustre: lustre-MDT0001: new disk, initializing [ 7876.118252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7876.154285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7876.159686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7879.489977] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7886.630213] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7889.797239] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7896.022337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.118629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7896.422299] Lustre: lustre-OST0000: new disk, initializing [ 7896.430285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7896.522495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7901.400173] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7906.332499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7906.341040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7906.414599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7907.801807] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7910.933421] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7911.123238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7914.579551] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7914.749028] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7916.514767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7916.517112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.521739] LustreError: Skipped 1 previous similar message [ 7916.522044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7916.553304] Lustre: Skipped 1 previous similar message [ 7921.639941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7921.642895] Lustre: Skipped 1 previous similar message [ 7922.363636] LustreError: 222562:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7922.470468] Lustre: server umount lustre-OST0000 complete [ 7926.752829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.754627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7926.758695] Lustre: Skipped 1 previous similar message [ 7926.767266] Lustre: Skipped 2 previous similar messages [ 7931.875380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7931.879646] Lustre: Skipped 1 previous similar message [ 7932.230248] LustreError: 222764:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7932.261043] LustreError: 222764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7932.367094] Lustre: server umount lustre-MDT0000 complete [ 7936.446817] LustreError: 219616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348639 with bad export cookie 5780233836228682606 [ 7936.449669] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7936.472765] LustreError: 219616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7936.694026] LustreError: 222966:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7936.746585] LustreError: 222966:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7937.071398] Lustre: server umount lustre-MDT0001 complete [ 7956.796888] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7965.261682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.803868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.961808] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7971.297888] LustreError: 224142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.311884] LustreError: 224142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7975.812842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7976.103937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7979.274692] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7982.228237] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7988.075864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7988.368658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.441057] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8001.443587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8001.504092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8001.651697] Lustre: lustre-OST0001: new disk, initializing [ 8001.656153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8007.198293] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8009.800940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8009.813959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8009.894317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8017.545568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8026.776935] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8041.967658] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:05:43 (1763348743) [ 8046.050517] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8046.053601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8046.061209] Lustre: Skipped 2 previous similar messages [ 8046.064081] Lustre: Skipped 1 previous similar message [ 8049.645589] LustreError: 227539:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8049.652547] LustreError: 227539:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8049.731689] Lustre: server umount lustre-MDT0000 complete [ 8051.172676] LustreError: 224142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8051.190560] LustreError: 224142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8052.685422] LustreError: 224121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348755 with bad export cookie 5780233836228683488 [ 8052.688706] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8052.699668] LustreError: 224121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8052.985512] Lustre: server umount lustre-MDT0001 complete [ 8066.584455] LustreError: 227940:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8066.588039] LustreError: 227940:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8066.627376] Lustre: server umount lustre-OST0000 complete [ 8080.208800] LustreError: 228142:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8080.214986] LustreError: 228142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8080.328264] Lustre: server umount lustre-OST0001 complete [ 8085.593796] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 8093.766858] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 8101.360167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8107.491978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8113.853254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8119.890883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8126.079266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8126.144360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.301584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8126.329418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8126.382950] Lustre: lustre-MDT0000: new disk, initializing [ 8126.456718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.461968] Lustre: Skipped 1 previous similar message [ 8126.484785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8130.745951] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8141.480781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8141.567857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8141.707685] Lustre: 230332:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8141.800901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8141.804595] Lustre: Skipped 1 previous similar message [ 8141.959294] Lustre: lustre-MDT0001: new disk, initializing [ 8142.091447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8142.097615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8145.828951] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8152.084824] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8155.451705] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8161.081067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8161.161187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8161.327124] Lustre: lustre-OST0000: new disk, initializing [ 8161.330641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8161.407229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8161.416336] Lustre: Skipped 1 previous similar message [ 8162.598965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8162.605893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8162.744298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8165.966237] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8172.732959] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8176.497910] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8176.702355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8180.200429] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8180.379775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8183.271425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.272766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8183.282385] Lustre: Skipped 1 previous similar message [ 8183.285142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8183.294668] Lustre: Skipped 3 previous similar messages [ 8188.159570] LustreError: 233272:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8188.166890] LustreError: 233272:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8188.315794] Lustre: server umount lustre-OST0000 complete [ 8193.506737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8193.506800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8193.526620] Lustre: Skipped 1 previous similar message [ 8193.531785] LustreError: Skipped 1 previous similar message [ 8202.721967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8202.724981] Lustre: Skipped 5 previous similar messages [ 8206.303285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8206.442473] Lustre: server umount lustre-MDT0000 complete [ 8207.841444] LustreError: 231167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.855664] LustreError: 231167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8209.522454] LustreError: 230324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348912 with bad export cookie 5780233836228685630 [ 8209.523222] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8209.538926] LustreError: 230324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8209.894436] Lustre: server umount lustre-MDT0001 complete [ 8217.908946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8218.261220] LustreError: 234155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8222.391768] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8223.715674] LustreError: 234155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.942205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8232.377617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8232.385889] Lustre: Skipped 1 previous similar message [ 8236.223108] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8241.545598] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8245.475954] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8252.315660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8257.965962] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8262.527312] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8406.500541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.501700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8406.515138] Lustre: Skipped 2 previous similar messages [ 8406.515560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8406.528434] Lustre: Skipped 1 previous similar message [ 8412.418966] LustreError: 236441:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8412.428967] LustreError: 236441:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8412.561158] Lustre: server umount lustre-OST0000 complete [ 8416.748864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8421.572167] Lustre: server umount lustre-MDT0000 complete [ 8421.856733] LustreError: 234154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8421.861321] LustreError: 234154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8424.706820] LustreError: 234135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349128 with bad export cookie 5780233836228686512 [ 8424.719238] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8424.984635] Lustre: server umount lustre-MDT0001 complete [ 8529.595556] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 8532.151803] Key type lgssc unregistered [ 8532.423299] LNet: 237415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8532.441842] LNetError: 237415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8532.476736] LNet: Removed LNI 192.168.201.141@tcp [ 8533.199171] Key type .llcrypt unregistered [ 8533.200858] Key type ._llcrypt unregistered [ 8556.794646] Key type ._llcrypt registered [ 8556.796252] Key type .llcrypt registered [ 8556.884469] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 8569.900484] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 8570.619977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8570.648723] alg: No test for adler32 (adler32-zlib) [ 8571.696433] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 8571.888833] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 8573.599170] Key type lgssc registered [ 8574.774738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8582.373739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8587.604958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8592.672081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8598.464630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8612.431860] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 8621.507363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8621.582099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8621.596536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.753127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8622.783925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8622.835736] Lustre: lustre-MDT0000: new disk, initializing [ 8622.911279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8622.940143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8625.750062] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8634.048796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8634.106391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8634.191872] Lustre: 241441:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8634.246281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8634.249244] Lustre: Skipped 1 previous similar message [ 8634.303172] Lustre: lustre-MDT0001: new disk, initializing [ 8634.387152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8634.434126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8634.444098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8637.747691] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8642.111874] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8647.379453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8647.445882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8647.655303] Lustre: lustre-OST0000: new disk, initializing [ 8647.667638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8647.781518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.204845] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8656.954045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8656.960633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8657.026082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8660.978065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8661.046522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8661.195947] Lustre: lustre-OST0001: new disk, initializing [ 8661.201520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8661.278248] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8664.092746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8664.099195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8664.181084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8665.619308] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8676.299642] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8683.195552] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8699.878307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.882449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8699.884211] Lustre: Skipped 3 previous similar messages [ 8699.895029] Lustre: Skipped 2 previous similar messages [ 8703.005758] LustreError: 244966:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8703.074880] Lustre: server umount lustre-MDT0000 complete [ 8705.913827] LustreError: 242223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349409 with bad export cookie 6468929720766446147 [ 8705.915226] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8705.922050] LustreError: 242223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8706.072573] LustreError: 245167:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8706.075944] LustreError: 245167:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8706.197947] Lustre: server umount lustre-MDT0001 complete [ 8719.370164] LustreError: 245369:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8719.389813] LustreError: 245369:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8719.448857] Lustre: server umount lustre-OST0000 complete [ 8733.727714] LustreError: 245571:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8733.759519] LustreError: 245571:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8733.890869] Lustre: server umount lustre-OST0001 complete [ 8746.465246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8756.676933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8757.671627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8765.478204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8774.050045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8775.064383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8792.266775] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 8802.781515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8802.804327] Lustre: lustre-MDT0000: reset Object Index mappings [ 8803.213392] LustreError: 248886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8803.235705] LustreError: 248886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8803.333935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.301847] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8808.420121] LustreError: 248886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8813.544332] LustreError: 248886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.870191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8815.882424] Lustre: lustre-MDT0001: reset Object Index mappings [ 8816.220253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8819.773390] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8822.351983] Lustre: 249996:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8828.566379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8834.540486] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8841.344474] LustreError: 250352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8842.153134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8842.372802] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8842.394473] Lustre: Skipped 1 previous similar message [ 8847.818900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8847.821115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8848.159684] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8855.503591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8859.014526] Lustre: 251842:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8868.323110] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.330525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8868.336027] Lustre: Skipped 1 previous similar message [ 8868.351759] Lustre: Skipped 1 previous similar message [ 8870.056624] LustreError: 251947:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8870.067655] LustreError: 251947:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8870.157443] Lustre: server umount lustre-MDT0000 complete [ 8873.460489] LustreError: 249603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8873.481644] LustreError: 249603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8873.812771] LustreError: 251860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349577 with bad export cookie 6468929720766452937 [ 8873.814780] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8873.825950] LustreError: 251860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8874.116739] Lustre: server umount lustre-MDT0001 complete [ 8887.843578] LustreError: 252350:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8887.851266] LustreError: 252350:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8887.930334] Lustre: server umount lustre-OST0000 complete [ 8901.100044] Lustre: server umount lustre-OST0001 complete [ 8917.545539] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:20:18 (1763349618) [ 8931.297394] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 8940.377959] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 8948.823864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8954.454218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8960.063158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8965.260628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8970.501612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8970.547744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.706866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8970.743142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8970.812780] Lustre: lustre-MDT0000: new disk, initializing [ 8970.898414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8970.920974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8974.316701] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8983.269103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8983.330918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8983.434212] Lustre: 256081:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8983.505370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8983.511034] Lustre: Skipped 1 previous similar message [ 8983.615585] Lustre: lustre-MDT0001: new disk, initializing [ 8983.782492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8983.796387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8986.957553] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8992.475559] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8995.441767] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9000.439890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9000.489730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9000.684426] Lustre: lustre-OST0000: new disk, initializing [ 9000.687702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9001.783364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9001.791493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9001.894195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9004.908072] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9011.107399] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9014.674862] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9014.911048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9018.406518] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9018.605722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9022.438215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9022.442104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9022.451918] Lustre: Skipped 2 previous similar messages [ 9022.456655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9022.472725] Lustre: Skipped 4 previous similar messages [ 9026.369098] LustreError: 259019:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9026.376442] LustreError: 259019:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 9026.511538] Lustre: server umount lustre-OST0000 complete [ 9030.116803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9030.123140] LustreError: Skipped 1 previous similar message [ 9030.133040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.143729] Lustre: Skipped 1 previous similar message [ 9030.149109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9035.568988] Lustre: server umount lustre-MDT0000 complete [ 9037.798026] LustreError: 256091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.810735] LustreError: 256091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9038.549362] LustreError: 257824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349741 with bad export cookie 6468929720766455030 [ 9038.550857] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9038.557279] LustreError: 257824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9038.907799] Lustre: server umount lustre-MDT0001 complete [ 9053.737631] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 9061.456433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.746715] LustreError: 260597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.794090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9061.796971] Lustre: Skipped 2 previous similar messages [ 9064.786262] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9072.110346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9076.045771] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9078.538809] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9084.068668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9088.884945] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9095.318921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9095.371883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9095.480304] Lustre: lustre-OST0001: new disk, initializing [ 9095.484167] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9097.494796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9097.505614] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9097.588103] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9100.442679] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9109.571354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9112.511341] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9121.898296] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9122.026177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9128.476694] LustreError: 264101:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9128.484210] LustreError: 264101:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9128.930806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9128.943226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9128.968177] Lustre: Skipped 1 previous similar message [ 9128.971111] LustreError: 263619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9128.980778] LustreError: 263619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9130.585887] Lustre: server umount lustre-MDT0000 complete [ 9133.544961] LustreError: 260578:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349836 with bad export cookie 6468929720766455912 [ 9133.549174] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9133.553547] LustreError: 260578:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9133.799191] Lustre: server umount lustre-MDT0001 complete [ 9147.562698] Lustre: server umount lustre-OST0000 complete [ 9150.431251] Lustre: 238954:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349837/real 1763349837] req@ffff9d4201c56d80 x1849005729601024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763349853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9150.498703] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.504652] Lustre: Skipped 3 previous similar messages [ 9152.030659] Lustre: server umount lustre-OST0001 complete [ 9170.507441] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 9179.879366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.951698] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9191.703815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9192.113451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9192.124177] Lustre: Skipped 4 previous similar messages [ 9195.787193] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9198.396435] Lustre: 266988:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9204.540271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9204.713253] LustreError: 267345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.733570] LustreError: 267345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9206.613842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9210.570889] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9218.537924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9220.224085] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9223.566860] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9229.968598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9233.226754] Lustre: 268845:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9242.569678] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9242.615647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9252.078720] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:25:53 (1763349953) [ 9265.631410] LustreError: 269552:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9265.633954] LustreError: 269552:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 9265.701252] Lustre: server umount lustre-MDT0000 complete [ 9268.435314] LustreError: 265862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349971 with bad export cookie 6468929720766462191 [ 9268.440536] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9270.241737] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9270.251372] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9270.263853] Lustre: Skipped 3 previous similar messages [ 9270.271882] Lustre: Skipped 3 previous similar messages [ 9274.744611] Lustre: server umount lustre-MDT0001 complete [ 9292.255131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9292.387213] Lustre: server umount lustre-OST0000 complete [ 9295.340800] Lustre: server umount lustre-OST0001 complete [ 9300.290494] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 9309.008471] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 9318.598150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9324.843969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9330.768805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9337.292511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9351.051837] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 9360.726047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9360.794156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9361.057170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9361.113594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9361.185254] Lustre: lustre-MDT0000: new disk, initializing [ 9361.297291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9364.709847] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9372.912458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9372.981107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9373.075508] Lustre: 273044:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9373.417974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9376.637218] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9385.485454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9385.531934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9387.821089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9389.955222] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9398.940759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9399.019130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9400.490181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9404.186835] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9412.887064] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9416.087553] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9416.094285] Lustre: Skipped 1 previous similar message [ 9431.828794] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 22:28:53 (1763350133) [ 9435.105804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9435.126721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.145049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9435.156249] Lustre: Skipped 2 previous similar messages [ 9439.566330] LustreError: 276654:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9439.578774] LustreError: 276654:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 9439.673789] Lustre: server umount lustre-MDT0000 complete [ 9441.763878] LustreError: 276168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9441.782091] LustreError: 276168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 9442.878179] LustreError: 273037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350146 with bad export cookie 6468929720766470066 [ 9442.880584] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9442.897348] LustreError: 273037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9443.158653] Lustre: server umount lustre-MDT0001 complete [ 9456.191904] Lustre: server umount lustre-OST0000 complete [ 9469.613039] Lustre: server umount lustre-OST0001 complete [ 9485.668980] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 9494.771933] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 9502.809917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9508.782418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9514.195815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9520.419909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9527.311047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9527.370694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.541912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9527.569679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9527.572559] Lustre: Skipped 4 previous similar messages [ 9527.619519] Lustre: lustre-MDT0000: new disk, initializing [ 9527.622442] Lustre: Skipped 3 previous similar messages [ 9527.723171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.727409] Lustre: Skipped 6 previous similar messages [ 9527.740521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9527.744506] Lustre: Skipped 3 previous similar messages [ 9531.071173] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9540.446644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9540.513260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9540.570375] Lustre: 280104:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9540.788349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9540.795032] Lustre: Skipped 2 previous similar messages [ 9543.823249] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9550.078996] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9553.424871] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9560.074465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9560.163512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9562.252285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9566.645682] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9573.663284] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9575.398844] Lustre: Mounted lustre-client [ 9576.266162] LustreError: 282609:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9d4201a4c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9576.341417] Lustre: Unmounted lustre-client [ 9585.641405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9585.647234] Lustre: Skipped 4 previous similar messages [ 9585.647992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9585.652324] Lustre: Skipped 5 previous similar messages [ 9590.513561] Lustre: server umount lustre-OST0000 complete [ 9600.919772] Lustre: server umount lustre-MDT0000 complete [ 9604.859768] LustreError: 280097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350308 with bad export cookie 6468929720766471592 [ 9604.869404] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9605.349732] Lustre: server umount lustre-MDT0001 complete [ 9615.335606] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 9617.580292] Key type lgssc unregistered [ 9637.215255] LNet: 1 local NIs in recovery (showing 1): 192.168.201.141@tcp [ 9637.349579] LNet: 283888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9637.363552] LNetError: 283888:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9704.941609] LNet: Removed LNI 192.168.201.141@tcp [ 9705.628190] Key type .llcrypt unregistered [ 9705.630111] Key type ._llcrypt unregistered [ 9721.413090] Key type ._llcrypt registered [ 9721.414583] Key type .llcrypt registered [ 9721.513361] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 9737.850413] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 9738.735775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9738.767812] alg: No test for adler32 (adler32-zlib) [ 9739.855651] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 9740.212115] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 9742.007172] Key type lgssc registered [ 9743.030696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9752.181218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9758.713523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9764.452624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9770.376850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9772.221849] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:34:33 (1763350473) [ 9778.583608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9778.621231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9778.632350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.867477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9779.916947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9779.965489] Lustre: lustre-MDT0000: new disk, initializing [ 9780.028946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9780.053651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9783.550309] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9792.390896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9792.445674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9792.521238] Lustre: 287083:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9792.567694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9792.571496] Lustre: Skipped 1 previous similar message [ 9792.626277] Lustre: lustre-MDT0001: new disk, initializing [ 9792.699025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9792.731093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9792.739606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9795.621309] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9802.012465] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9805.773538] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9812.330610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9812.389253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9812.602702] Lustre: lustre-OST0000: new disk, initializing [ 9812.610092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9812.684991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9817.273536] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 9821.269539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9821.285605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9821.443469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9823.959114] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9831.906130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9831.908605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9831.910587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9831.923208] Lustre: Skipped 1 previous similar message [ 9833.278641] LustreError: 289524:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9833.374527] Lustre: server umount lustre-OST0000 complete [ 9837.030122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9837.032025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9837.038351] Lustre: Skipped 1 previous similar message [ 9837.045737] Lustre: Skipped 1 previous similar message [ 9842.145053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9842.719405] LustreError: 289726:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9842.722660] LustreError: 289726:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9842.792350] Lustre: server umount lustre-MDT0000 complete [ 9845.533138] LustreError: 287074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350548 with bad export cookie 4373649308662446743 [ 9845.533514] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9845.542346] LustreError: 287074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9845.652518] LustreError: 289928:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9845.659103] LustreError: 289928:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9845.810122] Lustre: server umount lustre-MDT0001 complete [ 9855.128727] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 9857.434500] Key type lgssc unregistered [ 9857.697867] LNet: 290460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9857.713259] LNetError: 290460:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9857.724675] LNet: Removed LNI 192.168.201.141@tcp [ 9858.512826] Key type .llcrypt unregistered [ 9858.521096] Key type ._llcrypt unregistered [ 9944.376337] Key type ._llcrypt registered [ 9944.377964] Key type .llcrypt registered [ 9944.528939] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 9944.690273] Key type .llcrypt unregistered [ 9944.692124] Key type ._llcrypt unregistered [10013.037831] Key type ._llcrypt registered [10013.042975] Key type .llcrypt registered [10013.175363] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10013.442622] Key type .llcrypt unregistered [10013.447772] Key type ._llcrypt unregistered [10041.027198] Key type ._llcrypt registered [10041.029013] Key type .llcrypt registered [10041.224393] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10041.594466] Key type .llcrypt unregistered [10041.596112] Key type ._llcrypt unregistered [10070.535123] Key type ._llcrypt registered [10070.537022] Key type .llcrypt registered [10070.652409] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10070.958371] Key type .llcrypt unregistered [10070.962073] Key type ._llcrypt unregistered [10145.963644] Key type ._llcrypt registered [10145.965309] Key type .llcrypt registered [10146.050992] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10146.338324] Key type .llcrypt unregistered [10146.342416] Key type ._llcrypt unregistered [10175.914234] Key type ._llcrypt registered [10175.916798] Key type .llcrypt registered [10176.008112] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10176.384434] Key type .llcrypt unregistered [10176.391040] Key type ._llcrypt unregistered [10211.208702] Key type ._llcrypt registered [10211.210951] Key type .llcrypt registered [10211.311404] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10211.596957] Key type .llcrypt unregistered [10211.598890] Key type ._llcrypt unregistered [10238.123898] Key type ._llcrypt registered [10238.127381] Key type .llcrypt registered [10238.216267] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10238.423904] Key type .llcrypt unregistered [10238.425984] Key type ._llcrypt unregistered [10266.576901] Key type ._llcrypt registered [10266.578385] Key type .llcrypt registered [10266.699139] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10266.999425] Key type .llcrypt unregistered [10267.007098] Key type ._llcrypt unregistered [10296.084879] Key type ._llcrypt registered [10296.086551] Key type .llcrypt registered [10296.178527] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10296.391535] Key type .llcrypt unregistered [10296.394836] Key type ._llcrypt unregistered [10357.685708] Key type ._llcrypt registered [10357.694246] Key type .llcrypt registered [10357.823706] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10358.179369] Key type .llcrypt unregistered [10358.186121] Key type ._llcrypt unregistered [10382.704255] Key type ._llcrypt registered [10382.706021] Key type .llcrypt registered [10382.798591] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10383.075410] Key type .llcrypt unregistered [10383.076901] Key type ._llcrypt unregistered [10413.258810] Key type ._llcrypt registered [10413.260716] Key type .llcrypt registered [10413.333708] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10413.587473] Key type .llcrypt unregistered [10413.589552] Key type ._llcrypt unregistered [10465.651884] Key type ._llcrypt registered [10465.653912] Key type .llcrypt registered [10465.756128] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [10483.409238] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [10484.180322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10484.192781] alg: No test for adler32 (adler32-zlib) [10485.276560] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [10485.477955] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [10487.143191] Key type lgssc registered [10488.112627] Lustre: Echo OBD driver; http://www.lustre.org/ [10496.881231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10503.155625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10508.437132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10514.252865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10515.930709] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10311 sec ========= 22:46:57 (1763351217) [10517.585720] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:46:58 (1763351218) === [10521.229550] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:47:02 (1763351222) === [10563.279355] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [10565.912040] Key type lgssc unregistered [10566.132960] LNet: 312512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10566.141795] LNetError: 312512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10566.158586] LNet: Removed LNI 192.168.201.141@tcp [10566.927186] Key type .llcrypt unregistered [10566.929735] Key type ._llcrypt unregistered