[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 419054808 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002227] x2apic enabled [ 0.003022] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006900] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009105] LSM: Security Framework initializing [ 0.010032] Yama: becoming mindful. [ 0.012017] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.020579] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025016] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028126] *** VALIDATE tmpfs *** [ 0.030252] *** VALIDATE proc *** [ 0.031262] *** VALIDATE cgroup *** [ 0.032004] *** VALIDATE cgroup2 *** [ 0.033249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034118] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036024] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040133] debug: unmapping init [mem 0xffffffffa6a59000-0xffffffffa6a60fff] [ 0.042147] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043599] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044015] ... version: 2 [ 0.045006] ... bit width: 48 [ 0.046006] ... generic registers: 4 [ 0.047007] ... value mask: 0000ffffffffffff [ 0.048006] ... max period: 00007fffffffffff [ 0.049006] ... fixed-purpose events: 3 [ 0.050005] ... event mask: 000000070000000f [ 0.051281] rcu: Hierarchical SRCU implementation. [ 0.053234] smp: Bringing up secondary CPUs ... [ 0.054478] x86: Booting SMP configuration: [ 0.055019] .... node #0, CPUs: #1 #2 #3 [ 0.058346] smp: Brought up 1 node, 4 CPUs [ 0.060010] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.209845] node 0 deferred pages initialised in 146ms [ 0.212121] devtmpfs: initialized [ 0.213282] x86/mm: Memory block size: 128MB [ 0.215831] gcov: version magic: 0x41383552 [ 0.219329] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.220069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.221332] pinctrl core: initialized pinctrl subsystem [ 0.222240] [ 0.222984] ************************************************************* [ 0.223009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.224008] ** ** [ 0.225012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.226018] ** ** [ 0.227016] ** This means that this kernel is built to expose internal ** [ 0.228009] ** IOMMU data structures, which may compromise security on ** [ 0.229008] ** your system. ** [ 0.230018] ** ** [ 0.231010] ** If you see this message and you are not debugging the ** [ 0.232010] ** kernel, report this immediately to your vendor! ** [ 0.233009] ** ** [ 0.234007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.235007] ************************************************************* [ 0.237000] NET: Registered protocol family 16 [ 0.237458] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.241049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.245058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.249013] cpuidle: using governor menu [ 0.250000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.251359] PCI: Using configuration type 1 for base access [ 0.253129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.262360] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.265041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.268202] cryptd: max_cpu_qlen set to 1000 [ 0.271259] ACPI: Added _OSI(Module Device) [ 0.273016] ACPI: Added _OSI(Processor Device) [ 0.275009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.277009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280645] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.287924] ACPI: Interpreter enabled [ 0.290098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.292021] ACPI: Using IOAPIC for interrupt routing [ 0.294230] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.298365] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.310612] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.312022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.314013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.315000] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.320334] acpiphp: Slot [2] registered [ 0.322147] acpiphp: Slot [5] registered [ 0.323100] acpiphp: Slot [6] registered [ 0.325138] acpiphp: Slot [7] registered [ 0.327167] acpiphp: Slot [8] registered [ 0.329187] acpiphp: Slot [9] registered [ 0.330092] acpiphp: Slot [10] registered [ 0.332101] acpiphp: Slot [3] registered [ 0.334062] acpiphp: Slot [4] registered [ 0.335077] acpiphp: Slot [11] registered [ 0.337068] acpiphp: Slot [12] registered [ 0.338057] acpiphp: Slot [13] registered [ 0.339109] acpiphp: Slot [14] registered [ 0.340058] acpiphp: Slot [15] registered [ 0.341057] acpiphp: Slot [16] registered [ 0.342059] acpiphp: Slot [17] registered [ 0.343167] acpiphp: Slot [18] registered [ 0.345178] acpiphp: Slot [19] registered [ 0.347137] acpiphp: Slot [20] registered [ 0.349106] acpiphp: Slot [21] registered [ 0.351133] acpiphp: Slot [22] registered [ 0.352057] acpiphp: Slot [23] registered [ 0.353073] acpiphp: Slot [24] registered [ 0.355064] acpiphp: Slot [25] registered [ 0.356061] acpiphp: Slot [26] registered [ 0.357087] acpiphp: Slot [27] registered [ 0.358067] acpiphp: Slot [28] registered [ 0.359064] acpiphp: Slot [29] registered [ 0.360063] acpiphp: Slot [30] registered [ 0.361067] acpiphp: Slot [31] registered [ 0.363065] PCI host bridge to bus 0000:00 [ 0.364000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.364000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.366011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.368013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.371014] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.375028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.377214] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.381043] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.384000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.396017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.402743] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.404015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.406011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.409016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.411490] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.413727] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.416038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.418677] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.423011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.435016] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.438000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.443522] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.448018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.454020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.472032] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.483933] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.491017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.499019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.517021] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.527762] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.536014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.544022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.558018] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.567545] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.573035] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.581018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.597033] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.609912] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.618021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641014] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.653012] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.664013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.678017] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.688393] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.690315] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.693212] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.698173] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.700309] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.704197] iommu: Default domain type: Passthrough [ 0.708593] SCSI subsystem initialized [ 0.709090] ACPI: bus type USB registered [ 0.710095] usbcore: registered new interface driver usbfs [ 0.713043] usbcore: registered new interface driver hub [ 0.714059] usbcore: registered new device driver usb [ 0.716162] pps_core: LinuxPPS API ver. 1 registered [ 0.718010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.720069] PTP clock support registered [ 0.722295] EDAC MC: Ver: 3.0.0 [ 0.724131] PCI: Using ACPI for IRQ routing [ 0.727372] NetLabel: Initializing [ 0.729014] NetLabel: domain hash size = 128 [ 0.730007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.733096] NetLabel: unlabeled traffic allowed by default [ 0.735127] vgaarb: loaded [ 0.737290] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.739025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.743966] clocksource: Switched to clocksource kvm-clock [ 0.855760] VFS: Disk quotas dquot_6.6.0 [ 0.857084] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.858824] *** VALIDATE ramfs *** [ 0.859603] *** VALIDATE hugetlbfs *** [ 0.861569] pnp: PnP ACPI init [ 0.863904] pnp: PnP ACPI: found 6 devices [ 0.882673] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.885246] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.887568] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.889760] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.891454] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.893356] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.895503] NET: Registered protocol family 2 [ 0.898062] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.901959] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.905914] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.911017] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.914093] TCP: Hash tables configured (established 65536 bind 65536) [ 0.916869] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.919647] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.922488] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925209] NET: Registered protocol family 1 [ 0.928199] RPC: Registered named UNIX socket transport module. [ 0.930254] RPC: Registered udp transport module. [ 0.931857] RPC: Registered tcp transport module. [ 0.933497] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.935799] NET: Registered protocol family 44 [ 0.937309] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.939329] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.941307] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.943606] PCI: CLS 0 bytes, default 64 [ 0.945293] Unpacking initramfs... [ 2.573221] debug: unmapping init [mem 0xffff997a3cc54000-0xffff997a3ffbffff] [ 2.577293] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.579297] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.581817] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.137148] Initialise system trusted keyrings [ 3.138654] Key type blacklist registered [ 3.140892] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.150522] zbud: loaded [ 3.154192] *** VALIDATE nfs *** [ 3.155308] *** VALIDATE nfs4 *** [ 3.157091] pstore: using deflate compression [ 3.160754] Platform Keyring initialized [ 3.308581] NET: Registered protocol family 38 [ 3.312477] Key type asymmetric registered [ 3.314891] Asymmetric key parser 'x509' registered [ 3.317094] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.320347] io scheduler mq-deadline registered [ 3.322591] io scheduler kyber registered [ 3.327405] io scheduler bfq registered [ 3.329736] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.334271] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.338112] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.341288] ACPI: Power Button [PWRF] [ 3.437587] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.531919] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.750720] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.934573] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.246637] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.286388] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.340089] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.357293] Non-volatile memory driver v1.3 [ 4.362900] Linux agpgart interface v0.103 [ 4.404223] virtio_blk virtio1: [vda] 133040 512-byte logical blocks (68.1 MB/65.0 MiB) [ 4.408538] vda: detected capacity change from 0 to 68116480 [ 4.432192] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.435465] vdb: detected capacity change from 0 to 1073741824 [ 4.456425] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.459425] vdc: detected capacity change from 0 to 2621440000 [ 4.501056] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.505313] vdd: detected capacity change from 0 to 2621440000 [ 4.532724] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.536485] vde: detected capacity change from 0 to 4294967296 [ 4.568572] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.571753] vdf: detected capacity change from 0 to 4294967296 [ 4.600266] libphy: Fixed MDIO Bus: probed [ 4.610295] usbcore: registered new interface driver usbserial_generic [ 4.614819] usbserial: USB Serial support registered for generic [ 4.617964] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.629516] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.631938] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.634224] mousedev: PS/2 mouse device common for all mice [ 4.637913] rtc_cmos 00:05: RTC can wake from S4 [ 4.642601] rtc_cmos 00:05: registered as rtc0 [ 4.645556] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.648287] intel_pstate: CPU model not supported [ 4.652340] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.653688] hid: raw HID events driver (C) Jiri Kosina [ 4.664548] usbcore: registered new interface driver usbhid [ 4.670474] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.670660] usbhid: USB HID core driver [ 4.670970] drop_monitor: Initializing network drop monitor service [ 4.683400] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.686357] Initializing XFRM netlink socket [ 4.689059] NET: Registered protocol family 10 [ 4.702534] Segment Routing with IPv6 [ 4.704351] NET: Registered protocol family 17 [ 4.706423] mpls_gso: MPLS GSO support [ 4.710659] RAS: Correctable Errors collector initialized. [ 4.712817] AVX version of gcm_enc/dec engaged. [ 4.714453] AES CTR mode by8 optimization enabled [ 4.840164] sched_clock: Marking stable (4840143997, 0)->(5645310021, -805166024) [ 4.844198] registered taskstats version 1 [ 4.848654] Loading compiled-in X.509 certificates [ 4.850510] zswap: loaded using pool lzo/zbud [ 4.878836] Key type big_key registered [ 4.899364] Key type encrypted registered [ 4.901150] ima: No TPM chip found, activating TPM-bypass! [ 4.903531] ima: Allocated hash algorithm: sha1 [ 4.904741] ima: No architecture policies found [ 4.906602] evm: Initialising EVM extended attributes: [ 4.913129] evm: security.selinux [ 4.914181] evm: security.ima [ 4.915298] evm: security.capability [ 4.916610] evm: HMAC attrs: 0x1 [ 4.920238] rtc_cmos 00:05: setting system clock to 2025-08-31 21:30:12 UTC (1756675812) [ 4.927958] debug: unmapping init [mem 0xffffffffa7a03000-0xffffffffa7bfffff] [ 4.931076] debug: unmapping init [mem 0xffffffffa6782000-0xffffffffa6a58fff] [ 4.939175] Write protecting the kernel read-only data: 28672k [ 4.942543] debug: unmapping init [mem 0xffffffffa4e03000-0xffffffffa4ffffff] [ 4.945200] debug: unmapping init [mem 0xffffffffa5714000-0xffffffffa57fffff] [ 5.004865] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.021394] systemd[1]: Detected virtualization kvm. [ 5.023259] systemd[1]: Detected architecture x86-64. [ 5.024989] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.049919] systemd[1]: No hostname configured. [ 5.051462] systemd[1]: Set hostname to . [ 5.053597] random: systemd: uninitialized urandom read (16 bytes read) [ 5.055959] systemd[1]: Initializing machine ID from random generator. [ 5.301435] random: systemd: uninitialized urandom read (16 bytes read) [ 5.304159] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.307665] random: systemd: uninitialized urandom read (16 bytes read) [ 5.309695] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.313961] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.319317] device-mapper: uevent: version 1.0.3 [ 6.321462] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 7.214927] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.040965] virtio_net virtio0 ens2: renamed from eth0 [ 8.363996] scsi host0: ata_piix [ 8.506295] scsi host1: ata_piix [ 8.507662] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.509811] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.431717] random: crng init done [ 13.432952] random: 7 urandom warning(s) missed due to ratelimiting [ 13.613039] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.191928] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ 15.752649] hrtimer: interrupt took 3701718 ns [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.825328] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.259311] SELinux: Disabled at runtime. [ 17.346154] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.358937] systemd[1]: Detected virtualization kvm. [ 17.360738] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.196038] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.203651] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.216472] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.220713] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.226021] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.246795] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.259695] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK [0[ 18.441323] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.006968] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.591183] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.598310] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.896357] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.930953] EDAC sbridge: Ver: 1.1.2 [ 23.419174] Key type dns_resolver registered [ 24.034607] NFS: Registering the id_resolver key type [ 24.040767] Key type id_resolver registered [ 24.042665] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg110-server login: [ 48.678899] libcfs: loading out-of-tree module taints kernel. [ 48.687998] Key type ._llcrypt registered [ 48.689651] Key type .llcrypt registered [ 48.725129] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 55.802896] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 56.338770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.344262] alg: No test for adler32 (adler32-zlib) [ 57.287760] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 57.549695] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 59.144134] Key type lgssc registered [ 59.656738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 66.018511] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 67.102953] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.584212] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.043191] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.506554] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.322868] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 84.532956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.556168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 84.568200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.671242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 85.684304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 85.720481] Lustre: lustre-MDT0000: new disk, initializing [ 85.741653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.747148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 87.049304] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 92.342649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.378025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 92.411215] Lustre: 6484:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 92.425541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 92.428483] Lustre: Skipped 1 previous similar message [ 92.472290] Lustre: lustre-MDT0001: new disk, initializing [ 92.495749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 92.504528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 92.506963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 93.930594] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.190833] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 99.541234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.566902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.669824] Lustre: lustre-OST0000: new disk, initializing [ 99.672561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 99.690563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 101.612702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 101.615485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 101.642366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 101.674536] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.182910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.210352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 107.250148] Lustre: lustre-OST0001: new disk, initializing [ 107.253068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 107.269187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 107.948761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 107.953476] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 107.973089] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 109.240419] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.228157] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 118.867027] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 125.904702] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing check_logdir /tmp/testlogs/ [ 127.410088] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing yml_node [ 128.894612] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 129.750835] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 130.621296] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 131.154336] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 31 17:32:18 EDT 2025 [ 137.074416] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 137.564745] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 154.080810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 154.080955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.086087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 154.087317] Lustre: Skipped 3 previous similar messages [ 155.449897] LustreError: 12186:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 155.476397] Lustre: server umount lustre-MDT0000 complete [ 156.670579] LustreError: 9454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756675964 with bad export cookie 1510347825824860129 [ 156.672112] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 156.673743] LustreError: 9454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 159.200889] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 159.201118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 159.204831] Lustre: Skipped 4 previous similar messages [ 161.248527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 161.250094] Lustre: Skipped 2 previous similar messages [ 162.813060] LustreError: 12389:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 162.816100] LustreError: 12389:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 162.905112] Lustre: server umount lustre-MDT0001 complete [ 164.222052] LustreError: 12590:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 164.224196] LustreError: 12590:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 164.242685] Lustre: server umount lustre-OST0000 complete [ 165.563172] Lustre: server umount lustre-OST0001 complete [ 167.589161] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 170.601530] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 173.905806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.091153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.076982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.233430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.561000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.583866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.669865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.680799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.715871] Lustre: lustre-MDT0000: new disk, initializing [ 182.740190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.747052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.058065] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 188.059424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.086376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 188.118531] Lustre: 14969:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.223226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.437584] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 192.476266] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 193.820141] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 196.115460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.140891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 196.282854] Lustre: lustre-OST0000: new disk, initializing [ 196.284247] Lustre: Skipped 1 previous similar message [ 196.286158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.288475] Lustre: Skipped 2 previous similar messages [ 196.321452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.324983] Lustre: Skipped 1 previous similar message [ 198.149802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.151972] Lustre: Skipped 1 previous similar message [ 198.154050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.201951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.386992] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 201.563454] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 203.073902] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 203.162701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 204.582155] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 204.671361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 208.353847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 208.355841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.356130] LustreError: Skipped 1 previous similar message [ 208.359827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 208.361801] Lustre: Skipped 1 previous similar message [ 211.439930] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 211.442249] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 211.519591] Lustre: server umount lustre-OST0000 complete [ 213.473901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 213.474553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 213.479136] Lustre: Skipped 2 previous similar messages [ 223.713775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 223.715370] Lustre: Skipped 4 previous similar messages [ 227.296126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 227.327140] LustreError: 18110:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 227.330674] LustreError: 18110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 227.368978] Lustre: server umount lustre-MDT0000 complete [ 228.652392] LustreError: 14961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676036 with bad export cookie 1510347825824861648 [ 228.655774] LustreError: 14961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 228.655798] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 228.816567] Lustre: server umount lustre-MDT0001 complete [ 230.795276] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:33:57 (1756676037) [ 237.366635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.537330] LustreError: 19446:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 237.542311] LustreError: 19446:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 237.565132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.905026] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 241.775306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 243.236815] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 245.070690] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 246.285749] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 246.988045] Lustre: Failing over lustre-MDT0000 [ 246.991365] LustreError: 19476:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 246.995236] LustreError: 19476:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 247.000023] LustreError: 19476:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 247.060059] LustreError: 20787:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 247.061739] LustreError: 20787:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 247.088669] Lustre: server umount lustre-MDT0000 complete [ 247.266641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 247.266847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 247.273682] LustreError: 19441:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 247.279082] LustreError: 19441:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 252.385737] LustreError: 19464:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 252.390269] LustreError: 19464:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 257.505484] LustreError: 19464:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 257.510397] LustreError: 19464:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 260.089934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.131919] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 261.518618] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 264.721729] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 265.698679] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 265.701637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 265.707901] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 270.816891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 270.819150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.825726] Lustre: Skipped 1 previous similar message [ 270.828873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.056122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 281.084778] LustreError: 21766:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 281.087299] LustreError: 21766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 281.112076] Lustre: server umount lustre-MDT0000 complete [ 282.081657] LustreError: 19446:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.087632] LustreError: 19446:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 282.433652] LustreError: 20164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676090 with bad export cookie 1510347825824863055 [ 282.437713] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 282.439402] LustreError: 20164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 282.613738] Lustre: server umount lustre-MDT0001 complete [ 285.632784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 285.823914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 285.825629] Lustre: Skipped 2 previous similar messages [ 287.244532] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 290.139380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 291.614200] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 293.417443] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 294.698374] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 295.414044] Lustre: Failing over lustre-MDT0000 [ 295.500108] Lustre: server umount lustre-MDT0000 complete [ 299.553674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.554309] LustreError: 22441:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.557063] Lustre: Skipped 2 previous similar messages [ 299.560614] LustreError: 22441:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 308.540984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 308.589136] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.016857] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 313.284712] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 313.825802] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 313.828183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 313.832490] Lustre: Skipped 1 previous similar message [ 313.837665] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 318.945027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.947746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.949383] Lustre: Skipped 4 previous similar messages [ 321.474307] LustreError: 24767:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 321.476413] LustreError: 24767:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 321.506725] Lustre: server umount lustre-MDT0000 complete [ 322.772228] LustreError: 22426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676130 with bad export cookie 1510347825824863986 [ 322.774982] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.776050] LustreError: 22426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 322.915072] Lustre: server umount lustre-MDT0001 complete [ 325.987408] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:35:32 (1756676132) [ 331.356315] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 334.454328] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 338.390438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.701836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 342.766541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 344.875368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 348.314487] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 352.005489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 352.026172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.105474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 352.116342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 352.151280] Lustre: lustre-MDT0000: new disk, initializing [ 352.174975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.177956] Lustre: Skipped 2 previous similar messages [ 352.184313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 353.540782] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 357.714479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.739600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 357.772839] Lustre: 28432:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 357.906651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 359.337643] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 362.607809] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 363.809760] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 366.130936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 366.160070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 368.135937] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 368.161914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 368.168592] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 368.249386] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 371.425537] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 373.491586] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 378.339395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.340205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 378.345622] Lustre: Skipped 3 previous similar messages [ 378.347477] LustreError: Skipped 1 previous similar message [ 379.404597] Lustre: server umount lustre-OST0000 complete [ 383.458894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.462260] Lustre: Skipped 5 previous similar messages [ 395.232115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 395.265450] LustreError: 31268:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 395.267068] LustreError: 31268:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 395.295985] Lustre: server umount lustre-MDT0000 complete [ 396.545493] LustreError: 28422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676204 with bad export cookie 1510347825824864385 [ 396.550848] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 396.782909] Lustre: server umount lustre-MDT0001 complete [ 400.718915] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 401.851209] Key type lgssc unregistered [ 401.987460] LNet: 32001:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.989530] LNetError: 32001:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 401.998450] LNet: Removed LNI 192.168.201.110@tcp [ 402.356132] Key type .llcrypt unregistered [ 402.357842] Key type ._llcrypt unregistered [ 411.659808] Key type ._llcrypt registered [ 411.660778] Key type .llcrypt registered [ 411.702274] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 412.016790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 412.024340] alg: No test for adler32 (adler32-zlib) [ 412.884725] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 412.971981] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 414.560122] Key type lgssc registered [ 414.917117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 417.453033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 417.459949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.609141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.945556] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 422.979787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 423.114834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.514635] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 426.379077] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 427.600344] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 429.924928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 430.068581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.088287] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 433.128537] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 434.003929] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 454.237822] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:37:41 (1756676261) [ 454.927066] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 455.581280] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 457.763350] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:37:44 (1756676264) [ 460.727082] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:37:47 (1756676267) [ 461.115425] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 463.943063] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 463.945398] Lustre: Skipped 2 previous similar messages [ 464.519166] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:37:51 (1756676271) [ 471.789098] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 471.791398] Lustre: Skipped 10 previous similar messages [ 472.743433] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:37:59 (1756676279) [ 473.734473] Lustre: *** cfs_fail_loc=131b, val=2*** [ 475.039604] Lustre: *** cfs_fail_loc=131b, val=2*** [ 477.784135] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:38:04 (1756676284) [ 481.929480] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 481.931330] Lustre: Skipped 7 previous similar messages [ 482.488558] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:38:09 (1756676289) [ 489.440865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.442826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.446192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 491.491291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 491.491767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.491991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 492.524871] LustreError: 39930:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 492.581741] Lustre: server umount lustre-OST0000 complete [ 495.072840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 495.074993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.078449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.079942] Lustre: Skipped 1 previous similar message [ 501.729129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 501.730700] Lustre: Skipped 3 previous similar messages [ 506.849056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 506.850681] Lustre: Skipped 1 previous similar message [ 508.384143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 508.422410] LustreError: 40133:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 508.425202] LustreError: 40133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 508.453788] Lustre: server umount lustre-MDT0000 complete [ 509.700710] LustreError: 33352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676317 with bad export cookie 8896714038616993853 [ 509.703351] LustreError: 33352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 509.703864] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.774889] LustreError: 40335:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 509.776955] LustreError: 40335:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 509.860416] Lustre: server umount lustre-MDT0001 complete [ 513.547913] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 514.672511] Key type lgssc unregistered [ 514.810447] LNet: 40865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.812819] LNetError: 40865:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 514.820595] LNet: Removed LNI 192.168.201.110@tcp [ 515.146694] Key type .llcrypt unregistered [ 515.147682] Key type ._llcrypt unregistered [ 521.152603] Key type ._llcrypt registered [ 521.153631] Key type .llcrypt registered [ 521.189827] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 527.496584] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 527.822502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 527.865128] alg: No test for adler32 (adler32-zlib) [ 528.732106] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 528.810919] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 530.392151] Key type lgssc registered [ 530.761749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 533.845755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 536.118851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 538.208242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 540.414099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 542.724040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 542.743708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 542.749185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.829779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 543.842838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 543.880508] Lustre: lustre-MDT0000: new disk, initializing [ 543.907818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.916330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 545.212928] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 549.474148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 549.497292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 549.528292] Lustre: 43965:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 549.551065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 549.552486] Lustre: Skipped 1 previous similar message [ 549.590556] Lustre: lustre-MDT0001: new disk, initializing [ 549.624989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 549.636264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 549.639197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 550.973823] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 554.158360] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 555.393662] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 557.659883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 557.678977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 557.774599] Lustre: lustre-OST0000: new disk, initializing [ 557.776364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 557.803586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.698568] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 562.774575] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 564.143136] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 565.750804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 565.753611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 565.781738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 566.262265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 567.631035] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 567.717494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 570.852090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.852377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.852978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.855137] LustreError: Skipped 1 previous similar message [ 570.858290] Lustre: Skipped 1 previous similar message [ 574.444111] LustreError: 46918:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 574.508066] Lustre: server umount lustre-OST0000 complete [ 575.969118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.973238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 575.975600] Lustre: Skipped 1 previous similar message [ 580.577032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.578841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.582102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 586.209082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 586.210945] Lustre: Skipped 3 previous similar messages [ 590.304120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 590.334453] LustreError: 47120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 590.336330] LustreError: 47120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 590.362816] Lustre: server umount lustre-MDT0000 complete [ 591.570181] LustreError: 43957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676399 with bad export cookie 4267083887933832891 [ 591.573292] LustreError: 43957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 591.573355] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.639587] LustreError: 47322:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 591.642633] LustreError: 47322:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 591.728855] Lustre: server umount lustre-MDT0001 complete [ 594.804581] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:40:01 (1756676401) [ 597.115862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.271438] LustreError: 48040:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.278519] LustreError: 48040:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 597.295717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.677178] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 601.754545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 601.896149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 603.370069] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 605.224508] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 606.526962] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 608.975168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 611.069197] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 612.940694] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 625.322169] LustreError: 50255:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 625.700474] Lustre: Modifying parameter general.timeout=1 in log params [ 626.832481] Lustre: Modifying parameter general.timeout=4 in log params [ 626.834300] Lustre: Skipped 2 previous similar messages [ 628.840211] Lustre: Modifying parameter general.timeout=11 in log params [ 628.843477] Lustre: Skipped 6 previous similar messages [ 633.011769] Lustre: Modifying parameter general.timeout=24 in log params [ 633.013373] Lustre: Skipped 12 previous similar messages [ 641.122811] Lustre: Modifying parameter general.timeout=51 in log params [ 641.124227] Lustre: Skipped 26 previous similar messages [ 657.234314] Lustre: Modifying parameter general.timeout=105 in log params [ 657.237206] Lustre: Skipped 53 previous similar messages [ 668.202811] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:41:15 (1756676475) [ 669.185394] Lustre: 57154:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 670.426766] Lustre: 57350:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 670.430451] Lustre: 57350:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 681.355752] Lustre: 58999:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 681.359164] Lustre: 58999:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 3 previous similar messages [ 681.361471] LustreError: 58999:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 682.779179] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:41:29 (1756676489) [ 688.508543] LustreError: 59695:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 688.510401] LustreError: 59695:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 688.538559] Lustre: server umount lustre-MDT0000 complete [ 689.121904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 689.124107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.127883] LustreError: 48039:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.765435] LustreError: 48022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676497 with bad export cookie 4267083887933833759 [ 689.769061] LustreError: 48022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 689.769421] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 691.170138] LustreError: 49656:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 691.170365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 694.177159] LustreError: 48035:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.869629] LustreError: 59896:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 695.872120] LustreError: 59896:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 695.941927] Lustre: server umount lustre-MDT0001 complete [ 712.160814] LustreError: 60097:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 712.162530] LustreError: 60097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 712.233273] Lustre: server umount lustre-OST0000 complete [ 715.032926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 716.299338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 717.557996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 718.835881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 721.147695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.236331] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 721.245734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 721.248824] Lustre: Skipped 47 previous similar messages [ 721.333304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 721.335921] Lustre: Skipped 1 previous similar message [ 722.602845] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 725.326772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 725.357129] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 725.365634] Lustre: 61546:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 726.770952] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 728.553618] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 729.804530] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 732.061964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 732.129672] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 734.145588] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 736.006079] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 741.637084] Lustre: 63779:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 748.003375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 748.004029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.009218] Lustre: Skipped 2 previous similar messages [ 748.011077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 748.012536] Lustre: Skipped 2 previous similar messages [ 749.547078] LustreError: 64024:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 749.548615] LustreError: 64024:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 749.606202] Lustre: server umount lustre-OST0000 complete [ 765.408163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 765.473514] Lustre: server umount lustre-MDT0000 complete [ 766.699139] LustreError: 61581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676574 with bad export cookie 4267083887933836034 [ 766.702112] LustreError: 61581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 766.702303] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.845450] Lustre: server umount lustre-MDT0001 complete [ 770.788218] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 771.955063] Key type lgssc unregistered [ 772.103508] LNet: 64967:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.106186] LNetError: 64967:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.113477] LNet: Removed LNI 192.168.201.110@tcp [ 772.468100] Key type .llcrypt unregistered [ 772.469175] Key type ._llcrypt unregistered [ 781.939133] Key type ._llcrypt registered [ 781.940182] Key type .llcrypt registered [ 781.979914] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 782.331064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 782.366056] alg: No test for adler32 (adler32-zlib) [ 783.232650] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 783.318227] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 784.896177] Key type lgssc registered [ 785.278954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.922402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 787.930185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.113895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.448186] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 793.418557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 793.620930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 794.995022] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 796.845883] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 798.067424] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 800.504286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 800.709913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 802.648321] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 804.434375] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 822.854140] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 829.409211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 829.411364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.414911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 830.436876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 830.438053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.441837] Lustre: Skipped 1 previous similar message [ 831.487963] LustreError: 69074:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 831.522592] Lustre: server umount lustre-MDT0000 complete [ 832.757300] LustreError: 66320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676640 with bad export cookie 7858875275116496512 [ 832.761881] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.829880] LustreError: 69277:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 832.832863] LustreError: 69277:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 832.913459] Lustre: server umount lustre-MDT0001 complete [ 843.440262] LustreError: 69477:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 843.442268] LustreError: 69477:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 843.518480] Lustre: server umount lustre-OST0000 complete [ 846.350226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 847.688598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 848.986133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.294993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.579691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.695482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 852.709958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 852.809405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.198446] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 857.233484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 857.267931] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 857.281428] Lustre: 70925:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 857.474990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 858.858514] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 860.715164] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 861.925853] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 864.245052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 864.319491] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 866.460669] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 868.299887] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 868.842309] Lustre: 73167:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 869.480306] Lustre: Modifying parameter general.jobid_var=disable in log params [ 872.929831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 872.932434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.934121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.936497] Lustre: Skipped 1 previous similar message [ 872.938780] Lustre: Skipped 2 previous similar messages [ 876.843098] LustreError: 73415:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 876.845018] LustreError: 73415:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 876.903122] Lustre: server umount lustre-OST0000 complete [ 883.169783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 883.169938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.170793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 883.170798] Lustre: Skipped 1 previous similar message [ 883.171788] LustreError: Skipped 1 previous similar message [ 883.174964] Lustre: Skipped 1 previous similar message [ 888.289177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 888.290639] Lustre: Skipped 2 previous similar messages [ 892.384177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 892.420302] LustreError: 73622:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 892.422630] LustreError: 73622:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 892.451544] Lustre: server umount lustre-MDT0000 complete [ 893.409616] LustreError: 70936:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.416662] LustreError: 70936:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 893.663114] LustreError: 72569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676701 with bad export cookie 7858875275116497324 [ 893.666284] LustreError: 72569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 893.666457] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.814862] Lustre: server umount lustre-MDT0001 complete [ 897.646622] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 898.732989] Key type lgssc unregistered [ 898.871471] LNet: 74355:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.874279] LNetError: 74355:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.882589] LNet: Removed LNI 192.168.201.110@tcp [ 899.199292] Key type .llcrypt unregistered [ 899.200250] Key type ._llcrypt unregistered [ 908.945558] Key type ._llcrypt registered [ 908.947178] Key type .llcrypt registered [ 908.984205] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 909.328099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.357779] alg: No test for adler32 (adler32-zlib) [ 910.216566] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 910.315943] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 911.904212] Key type lgssc registered [ 912.320896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.995153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.001964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.173099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.564569] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 920.534497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 920.725532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 922.113933] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 923.924646] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.156349] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 927.430834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 927.605563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.449879] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 931.280127] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 944.109788] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 944.704498] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 944.706793] Lustre: Skipped 1 previous similar message [ 945.886199] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 945.887958] Lustre: Skipped 3 previous similar messages [ 948.011016] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 948.012758] Lustre: Skipped 6 previous similar messages [ 952.089295] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 952.091142] Lustre: Skipped 13 previous similar messages [ 960.318651] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 960.320843] Lustre: Skipped 26 previous similar messages [ 978.237277] Lustre: 83485:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 978.600352] Lustre: Disabling parameter general.jobid_var= in log params [ 978.603089] Lustre: Skipped 45 previous similar messages [ 979.200889] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:46:26 (1756676786) [ 979.770297] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 980.333259] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:46:27 (1756676787) [ 981.985981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.987650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.990269] LustreError: Skipped 1 previous similar message [ 981.997679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 982.001200] Lustre: Skipped 1 previous similar message [ 987.105975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.107653] Lustre: Skipped 1 previous similar message [ 987.436179] LustreError: 83781:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 987.505405] Lustre: server umount lustre-OST0000 complete [ 992.226130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 992.226201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.233512] Lustre: Skipped 1 previous similar message [ 992.235433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.238243] Lustre: Skipped 1 previous similar message [ 997.344991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.346500] Lustre: Skipped 1 previous similar message [ 1002.465235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1002.467169] Lustre: Skipped 1 previous similar message [ 1002.976134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1003.014833] LustreError: 83982:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1003.016855] LustreError: 83982:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1003.041415] Lustre: server umount lustre-MDT0000 complete [ 1004.300689] LustreError: 75707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676811 with bad export cookie 17285286999390273769 [ 1004.305990] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1004.309199] LustreError: 75707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1004.370046] LustreError: 84188:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1004.372125] LustreError: 84188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1004.477887] Lustre: server umount lustre-MDT0001 complete [ 1008.298357] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1009.477199] Key type lgssc unregistered [ 1009.613354] LNet: 84718:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.615313] LNetError: 84718:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.624309] LNet: Removed LNI 192.168.201.110@tcp [ 1009.947718] Key type .llcrypt unregistered [ 1009.950442] Key type ._llcrypt unregistered [ 1011.384367] Key type ._llcrypt registered [ 1011.385399] Key type .llcrypt registered [ 1011.425019] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1012.747269] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules [ 1013.111774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1013.152857] alg: No test for adler32 (adler32-zlib) [ 1014.004305] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1014.088751] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1015.664171] Key type lgssc registered [ 1016.053664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1016.460945] LustreError: 85834:0:(lu_object.c:1390:lu_context_key_register()) cfs_race id 60d sleeping [ 1024.355119] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1027.560690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.726607] LustreError: 85834:0:(lu_object.c:1390:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1037.735075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.905833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.122850] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1043.011286] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:47:30 (1756676850) [ 1044.554037] LustreError: 86873:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1044.557441] LustreError: 86873:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1044.559746] LustreError: 86873:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1044.622982] LustreError: 87644:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1044.707345] Lustre: server umount lustre-MDT0000 complete [ 1049.368611] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1050.407253] Key type lgssc unregistered [ 1050.526420] LNet: 88329:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.530183] LNetError: 88329:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.537292] LNet: Removed LNI 192.168.201.110@tcp [ 1050.819298] Key type .llcrypt unregistered [ 1050.820259] Key type ._llcrypt unregistered [ 1057.966987] Key type ._llcrypt registered [ 1057.967918] Key type .llcrypt registered [ 1058.013441] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1058.402890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1058.440337] alg: No test for adler32 (adler32-zlib) [ 1059.293148] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1059.379880] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1060.960137] Key type lgssc registered [ 1061.306496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1063.896182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1063.903983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.080070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.466823] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1069.447204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1069.683353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.062244] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1072.937767] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1074.210666] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1076.606216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1076.801337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.714142] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1080.558430] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1112.103616] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:48:39 (1756676919) [ 1112.689398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1113.307078] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:48:40 (1756676920) [ 1115.616850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1115.620103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.625075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.129757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.131183] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.133175] Lustre: Skipped 1 previous similar message [ 1116.136443] Lustre: Skipped 1 previous similar message [ 1120.383915] LustreError: 92430:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1120.411685] Lustre: server umount lustre-MDT0000 complete [ 1121.249948] LustreError: 89505:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.258337] LustreError: 89505:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 1121.652060] LustreError: 89485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676929 with bad export cookie 4009725196286409303 [ 1121.655029] LustreError: 89485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1121.655603] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1121.717675] LustreError: 92633:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1121.720809] LustreError: 92633:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1121.795820] Lustre: server umount lustre-MDT0001 complete [ 1133.233953] LustreError: 92834:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1133.235606] LustreError: 92834:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1133.365230] Lustre: server umount lustre-OST0000 complete [ 1137.281305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.442328] LustreError: 93463:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.497556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.834262] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1141.740038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1142.006417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1143.440707] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1145.195317] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1146.528177] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1148.621330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1150.801288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1150.825427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1150.892662] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1150.897038] LustreError: 93452:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1150.908437] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1150.911127] LustreError: 95321:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -98 [ 1150.913249] LustreError: 95321:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 1150.914825] LustreError: 95321:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1150.963148] Lustre: server umount lustre-OST0000 complete [ 1150.964426] LustreError: 95321:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1153.790415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1153.854886] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1153.858027] LustreError: 93451:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1153.867173] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1153.870032] LustreError: 95787:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -98 [ 1153.872188] LustreError: 95787:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 1153.874088] LustreError: 95787:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1153.921647] Lustre: server umount lustre-OST0000 complete [ 1153.923135] LustreError: 95787:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1154.848738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1157.100695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1157.171152] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1157.173890] Lustre: Found index 0 for lustre-OST0000, updating log [ 1157.176740] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1157.200464] Lustre: lustre-OST0000: new disk, initializing [ 1157.203241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1157.297795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.191044] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1160.433553] Lustre: Failing over lustre-OST0000 [ 1160.450186] LustreError: 96869:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1160.452021] LustreError: 96869:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1160.521047] Lustre: server umount lustre-OST0000 complete [ 1162.723133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.723413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1162.724572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.724577] Lustre: Skipped 1 previous similar message [ 1162.728793] Lustre: Skipped 1 previous similar message [ 1166.817275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.820012] Lustre: Skipped 2 previous similar messages [ 1167.933420] LustreError: 97070:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1167.936383] LustreError: 97070:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1167.965809] Lustre: server umount lustre-MDT0000 complete [ 1169.242715] LustreError: 93443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676976 with bad export cookie 4009725196286412033 [ 1169.246859] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.247606] LustreError: 93443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1169.428665] Lustre: server umount lustre-MDT0001 complete [ 1172.387770] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:49:39 (1756676979) [ 1177.590261] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1181.508427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.648463] LustreError: 98635:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1181.679104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.026464] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1185.851361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1187.487196] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1188.463780] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1190.746532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1193.065544] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1195.948994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1195.975575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1195.998524] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1196.031718] Lustre: lustre-OST0001: new disk, initializing [ 1196.033389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1198.042457] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1198.344759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1198.347363] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1198.385827] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1201.302297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1204.988071] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1211.872943] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1211.875026] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.878162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1217.543916] LustreError: 101874:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1217.547266] LustreError: 101874:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1217.572616] Lustre: server umount lustre-MDT0001 complete [ 1218.517882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1219.041512] LustreError: 98635:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.045284] LustreError: 98635:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1220.731800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1220.761245] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1220.762942] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1220.776733] Lustre: 98625:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1221.019608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1221.022441] Lustre: Skipped 3 previous similar messages [ 1221.864429] LustreError: 102549:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1221.867890] LustreError: 102549:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1221.871216] LustreError: 98627:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1221.902025] LustreError: 102557:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.201.110@tcp: cfg command failed: rc = -17 [ 1221.904596] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1221.904596] [ 1222.342188] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1225.604029] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 17:50:32 (1756677032) [ 1226.216654] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1231.415583] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1237.395207] Lustre: 104479:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1245.422056] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1246.585602] Lustre: 105612:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1297.889795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.890019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1297.890407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1297.890414] Lustre: Skipped 4 previous similar messages [ 1297.893027] Lustre: Skipped 4 previous similar messages [ 1303.834087] LustreError: 106043:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1303.836404] LustreError: 106043:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1303.863082] Lustre: server umount lustre-MDT0000 complete [ 1305.199565] LustreError: 98615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677112 with bad export cookie 4009725196286412943 [ 1305.200855] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.202677] LustreError: 98615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1305.357729] Lustre: server umount lustre-MDT0001 complete [ 1317.386435] Lustre: server umount lustre-OST0000 complete [ 1328.952023] Lustre: server umount lustre-OST0001 complete [ 1330.991282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1334.069287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1334.403672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1337.449432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1340.557388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1340.956040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1347.560265] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1351.255646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.262910] Lustre: lustre-MDT0000: reset Object Index mappings [ 1351.417748] LustreError: 109524:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.424984] LustreError: 109524:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1351.445297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.019942] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1355.918780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1355.928055] Lustre: lustre-MDT0001: reset Object Index mappings [ 1357.597889] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1358.619432] Lustre: 110639:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1361.004800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1362.277621] LustreError: 110997:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.281066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 1363.227343] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1365.988961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1367.213946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 1367.215767] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1367.216367] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 1368.063164] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1371.257233] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1372.394465] Lustre: 112495:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1392.641959] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:53:19 (1756677199) [ 1421.795637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.798649] Lustre: Skipped 4 previous similar messages [ 1421.798899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1421.802327] Lustre: Skipped 10 previous similar messages [ 1436.128137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1436.168374] LustreError: 112846:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1436.170796] LustreError: 112846:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1436.196423] Lustre: server umount lustre-MDT0000 complete [ 1437.157291] LustreError: 110243:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.170557] LustreError: 110243:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1437.398135] LustreError: 111817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677244 with bad export cookie 4009725196286459710 [ 1437.402906] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.403619] LustreError: 111817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1442.273943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.277115] Lustre: Skipped 1 previous similar message [ 1443.731564] Lustre: server umount lustre-MDT0001 complete [ 1451.073096] Lustre: server umount lustre-OST0000 complete [ 1458.418584] Lustre: server umount lustre-OST0001 complete [ 1460.307440] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1463.422324] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1467.015125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1469.216823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1471.224463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1473.301476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1475.623640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1475.648785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.726523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1475.738365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1475.778291] Lustre: lustre-MDT0000: new disk, initializing [ 1475.806361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1477.077542] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1480.125765] Lustre: server umount lustre-MDT0000 complete [ 1481.125337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.401601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.426423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.519101] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1483.524153] Lustre: 116716:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1483.528386] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1483.601261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.603044] Lustre: Skipped 4 previous similar messages [ 1484.867620] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1486.509552] Lustre: server umount lustre-MDT0000 complete [ 1489.402247] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:54:56 (1756677296) [ 1489.937197] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1490.569745] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:54:57 (1756677297) [ 1496.352153] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1499.770727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.241307] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1504.105060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1504.131751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1504.170034] Lustre: 118733:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1504.192984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1504.198258] Lustre: Skipped 1 previous similar message [ 1504.237601] Lustre: lustre-MDT0001: new disk, initializing [ 1504.286174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1504.291227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1505.613358] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1507.909859] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1510.074238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.097985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1510.187177] Lustre: lustre-OST0000: new disk, initializing [ 1510.188850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1511.100023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1511.103505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1511.140850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1512.187416] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1516.326245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1516.343262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1518.278982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1518.281898] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1518.318800] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1518.378325] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1522.830089] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1524.060350] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1683.729984] Lustre: 127501:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1683.734499] Lustre: 127501:0:(osd_handler.c:2079:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1683.737763] Lustre: 127501:0:(osd_handler.c:2086:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1683.741242] Lustre: 127501:0:(osd_handler.c:2096:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1683.744848] Lustre: 127501:0:(osd_handler.c:2103:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1683.747852] Lustre: 127501:0:(osd_handler.c:2110:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1766.621133] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:59:33 (1756677573) [ 1780.704922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1780.707231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.711385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.713140] Lustre: Skipped 10 previous similar messages [ 1784.253863] LustreError: 127708:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1784.255445] LustreError: 127708:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1784.311761] Lustre: server umount lustre-MDT0000 complete [ 1785.459396] LustreError: 127529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677593 with bad export cookie 4009725196286513120 [ 1785.462493] LustreError: 127529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1785.463688] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1787.873094] LustreError: 121920:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.877836] LustreError: 121920:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1791.797960] Lustre: server umount lustre-MDT0001 complete [ 1799.559591] Lustre: server umount lustre-OST0000 complete [ 1806.962446] Lustre: server umount lustre-OST0001 complete [ 1809.191384] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1812.581137] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1816.131447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1818.437059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1820.484595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1822.627783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1824.876279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1824.896948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.980434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1824.991532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1824.993263] Lustre: Skipped 1 previous similar message [ 1825.030072] Lustre: lustre-MDT0000: new disk, initializing [ 1825.031272] Lustre: Skipped 1 previous similar message [ 1825.054841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.056774] Lustre: Skipped 4 previous similar messages [ 1825.062589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1826.332180] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1830.472929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1830.496369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1830.527834] Lustre: 130490:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1830.630659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1831.932284] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1835.068836] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.311527] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1838.585799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1838.613915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1840.671472] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1840.781168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1840.824877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1843.833158] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1846.992841] Lustre: lustre-MDD0000: changelog on [ 1847.888924] Lustre: lustre-MDD0001: changelog on [ 2217.733902] Lustre: lustre-MDD0001: changelog off [ 2218.344339] Lustre: 130496:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2218.650070] Lustre: lustre-MDD0000: changelog off [ 2223.011214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.012357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.014489] Lustre: Skipped 6 previous similar messages [ 2223.019279] Lustre: Skipped 9 previous similar messages [ 2226.223091] LustreError: 139731:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2226.224727] LustreError: 139731:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2226.317106] Lustre: server umount lustre-OST0000 complete [ 2233.740856] Lustre: server umount lustre-MDT0000 complete [ 2235.016366] LustreError: 131271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678042 with bad export cookie 4009725196287318351 [ 2235.020318] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2235.022746] LustreError: 131271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2235.195306] Lustre: server umount lustre-MDT0001 complete [ 2238.672875] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2239.773269] Key type lgssc unregistered [ 2239.915599] LNet: 140664:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.918397] LNetError: 140664:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.930550] LNet: Removed LNI 192.168.201.110@tcp [ 2240.282116] Key type .llcrypt unregistered [ 2240.283261] Key type ._llcrypt unregistered [ 2246.986778] Key type ._llcrypt registered [ 2246.987746] Key type .llcrypt registered [ 2247.038210] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 2253.190143] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2253.555495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2253.568612] alg: No test for adler32 (adler32-zlib) [ 2254.437615] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2254.541361] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2256.128208] Key type lgssc registered [ 2256.520127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2259.608524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2261.981145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2264.008451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2266.091350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2268.259960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2268.284270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2268.289789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.381743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2269.395476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2269.434807] Lustre: lustre-MDT0000: new disk, initializing [ 2269.458778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.464769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2270.800218] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2274.956638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2274.981200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2275.013071] Lustre: 143814:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2275.040923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2275.042926] Lustre: Skipped 1 previous similar message [ 2275.078504] Lustre: lustre-MDT0001: new disk, initializing [ 2275.119251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2275.130836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2275.136170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2276.541619] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2279.627218] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2280.830190] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2283.116201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2283.146700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2283.268932] Lustre: lustre-OST0000: new disk, initializing [ 2283.271710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2283.305563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.170993] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2288.183901] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2292.223060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2292.226252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2292.265735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2295.776693] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_module obdecho/obdecho [ 2296.402610] LustreError: 146432:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2296.406637] LustreError: 146432:0:(obd_config.c:782:class_setup()) setup ec failed (-22) [ 2296.986281] LustreError: 146528:0:(echo_client.c:1881:echo_md_handler()) Can not resolve the path : rc = -2 [ 2300.896904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2300.899889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.906237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.435672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.436526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.440701] Lustre: Skipped 1 previous similar message [ 2306.368299] LustreError: 146869:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2306.396626] Lustre: server umount lustre-MDT0000 complete [ 2307.608504] LustreError: 143804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678115 with bad export cookie 10779590988242772330 [ 2307.611722] LustreError: 143804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2307.613276] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2307.675540] LustreError: 147073:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2307.677222] LustreError: 147073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2307.764791] Lustre: server umount lustre-MDT0001 complete [ 2319.217917] LustreError: 147274:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2319.220615] LustreError: 147274:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2319.294822] Lustre: server umount lustre-OST0000 complete [ 2326.106912] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 2329.488399] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2332.811775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2334.986980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2337.091434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2339.231713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2341.612728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2341.639939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.741639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2341.753942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2341.795125] Lustre: lustre-MDT0000: new disk, initializing [ 2341.822582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.829543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2343.245420] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2347.404317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2347.422722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2347.451906] Lustre: 150217:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2347.482637] LustreError: 150228:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.485995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2347.488154] LustreError: 150228:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 2347.489525] Lustre: Skipped 1 previous similar message [ 2347.528316] Lustre: lustre-MDT0001: new disk, initializing [ 2347.562113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2347.575492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2347.580187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2348.877472] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2351.970748] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2353.173565] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2355.391174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2355.414171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2355.529061] Lustre: lustre-OST0000: new disk, initializing [ 2355.531504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2356.664476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2356.670267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2356.703351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2357.499757] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2360.644115] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2362.033297] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2362.101149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2363.463854] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2363.545606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2366.255502] LustreError: 153156:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2366.258756] LustreError: 153156:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2366.348737] Lustre: server umount lustre-OST0000 complete [ 2367.968978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2367.971142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.974933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2367.976560] Lustre: Skipped 2 previous similar messages [ 2372.065576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.065886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.185236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.186872] Lustre: Skipped 2 previous similar messages [ 2378.212500] LustreError: 153357:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2378.214222] LustreError: 153357:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2378.239294] Lustre: server umount lustre-MDT0000 complete [ 2379.534727] LustreError: 153359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678187 with bad export cookie 10779590988242773268 [ 2379.539033] LustreError: 153359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2379.539253] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.989759] LustreError: 153561:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2389.991961] LustreError: 153561:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2390.090048] Lustre: server umount lustre-MDT0001 complete [ 2391.448714] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:09:58 (1756678198) [ 2391.995799] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2392.568528] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:09:59 (1756678199) [ 2395.130038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.288561] LustreError: 154233:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.311615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.313312] Lustre: Skipped 1 previous similar message [ 2396.659334] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2399.530576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2401.038464] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2402.817299] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2403.978201] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2406.184320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2408.247444] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2410.095994] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2424.802422] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.803901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.806355] Lustre: Skipped 1 previous similar message [ 2428.612438] LustreError: 156498:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2428.615239] LustreError: 156498:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2428.646799] Lustre: server umount lustre-MDT0000 complete [ 2429.921923] LustreError: 154227:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.926163] LustreError: 154227:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2429.928829] LustreError: 154214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678237 with bad export cookie 10779590988242774129 [ 2429.933974] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.934533] LustreError: 154214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2435.040866] LustreError: 154227:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.041151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.045799] LustreError: 154227:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2435.048593] Lustre: Skipped 1 previous similar message [ 2440.161574] LustreError: 155586:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.169237] LustreError: 155586:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2441.099481] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [ 2441.102844] Lustre: Skipped 7 previous similar messages [ 2444.256122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2444.374966] Lustre: server umount lustre-MDT0001 complete [ 2446.560205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2450.934241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2451.037403] LustreError: 157759:0:(llog.c:1602:llog_backup()) MGC192.168.201.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2451.042559] Lustre: 157759:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2464.736994] LustreError: 157767:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.741536] LustreError: 157767:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2482.058669] LustreError: 157783:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.064523] LustreError: 157783:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 13 previous similar messages [ 2512.352358] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2512.354132] Lustre: 157759:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2515.425079] LustreError: 157784:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.429398] LustreError: 157784:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 22 previous similar messages [ 2541.024257] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2541.025864] Lustre: 154224:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2552.872758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2552.874842] Lustre: Skipped 2 previous similar messages [ 2554.211067] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2555.876572] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2557.152774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.504319] LustreError: 154224:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2575.328856] LustreError: 142027:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008c6509e5 x1842010455641984/t0(0) o250->MGC192.168.201.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2576.766124] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2578.449513] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2578.983189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2580.135942] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2580.458751] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2583.766946] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2590.527340] LustreError: 159244:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2590.529154] LustreError: 159244:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2590.559697] Lustre: server umount lustre-MDT0000 complete [ 2590.691261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2590.692681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.695203] LustreError: 157783:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.696297] Lustre: Skipped 2 previous similar messages [ 2590.703072] LustreError: 157783:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 46 previous similar messages [ 2591.809220] LustreError: 154212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678399 with bad export cookie 10779590988242780625 [ 2591.811751] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2591.812978] LustreError: 154212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2591.965985] Lustre: server umount lustre-MDT0001 complete [ 2603.512480] Lustre: server umount lustre-OST0000 complete [ 2610.351310] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 2613.285831] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2616.995076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2619.110557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2621.046052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2623.155351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2625.535591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2625.563301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.648570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2625.660038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2625.701050] Lustre: lustre-MDT0000: new disk, initializing [ 2625.727154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.729214] Lustre: Skipped 1 previous similar message [ 2625.736364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2627.032330] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2631.249756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2631.278872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2631.316288] Lustre: 162588:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2631.436068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2632.754991] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2635.831649] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2637.031122] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2639.257930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2639.282831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2641.439958] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2644.474046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2644.516831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2644.523904] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2645.836095] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2645.910307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2647.177449] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2647.264429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2649.571835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2649.571893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.572653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2649.574367] LustreError: Skipped 1 previous similar message [ 2649.577212] Lustre: Skipped 1 previous similar message [ 2654.063771] Lustre: server umount lustre-OST0000 complete [ 2657.248872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2661.443820] LustreError: 165727:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2661.447051] LustreError: 165727:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 2661.474143] Lustre: server umount lustre-MDT0000 complete [ 2662.765761] LustreError: 162579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678470 with bad export cookie 10779590988242797418 [ 2662.769293] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.770972] LustreError: 162579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2662.935088] Lustre: server umount lustre-MDT0001 complete [ 2666.017601] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:14:33 (1756678473) [ 2668.298460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.812488] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2672.699067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2674.098852] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2675.928683] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2677.108084] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2679.374289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2681.336098] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2683.154620] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2702.654823] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:15:09 (1756678509) [ 2704.353517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2709.765278] Lustre: server umount lustre-OST0000 complete [ 2714.081209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.085445] Lustre: Skipped 4 previous similar messages [ 2714.087287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2714.090189] Lustre: Skipped 8 previous similar messages [ 2717.268929] Lustre: server umount lustre-MDT0000 complete [ 2718.463667] LustreError: 168215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678526 with bad export cookie 10779590988242798286 [ 2718.467310] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.468661] LustreError: 168215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2718.622342] Lustre: server umount lustre-MDT0001 complete [ 2722.266144] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2723.257818] Key type lgssc unregistered [ 2723.386351] LNet: 169997:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2723.389553] LNetError: 169997:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2723.395343] LNet: Removed LNI 192.168.201.110@tcp [ 2723.686673] Key type .llcrypt unregistered [ 2723.688070] Key type ._llcrypt unregistered [ 2731.222121] Key type ._llcrypt registered [ 2731.223054] Key type .llcrypt registered [ 2731.257842] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2731.671467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.683445] alg: No test for adler32 (adler32-zlib) [ 2732.531331] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2732.617642] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2734.200165] Key type lgssc registered [ 2734.575579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2737.250944] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2737.258972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2754.784353] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2760.929529] LustreError: 171197:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.110@tcp: failed processing log, type 4: rc = -110 [ 2766.048493] LustreError: 171197:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2766.054866] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2766.058168] LustreError: MGC192.168.201.110@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2766.063834] LustreError: 171197:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2766.068583] LustreError: 171197:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -22 [ 2766.072642] LustreError: 171197:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 2766.076492] LustreError: 171197:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2766.139491] Lustre: server umount lustre-OST0000 complete [ 2766.140958] LustreError: 171197:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2769.037165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.205070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.548194] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2774.358872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2774.486312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.916829] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2777.793451] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2779.050856] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2781.415995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2781.496801] LustreError: 173289:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2781.500787] Lustre: 173289:0:(mgc_request_server.c:706:mgc_llog_local_copy()) MGC192.168.201.110@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2781.557334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2783.868798] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2785.748738] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2789.859819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2789.860126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.861524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2789.871735] Lustre: Skipped 1 previous similar message [ 2792.574701] LustreError: 173939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2792.577528] LustreError: 173939:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2792.605743] Lustre: server umount lustre-MDT0000 complete [ 2794.031658] LustreError: 171678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678601 with bad export cookie 2974988041838062452 [ 2794.036928] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.037130] LustreError: 171678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2794.106975] LustreError: 174142:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2794.109442] LustreError: 174142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2794.202111] Lustre: server umount lustre-MDT0001 complete [ 2806.002751] LustreError: 174343:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2806.004686] LustreError: 174343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2806.090755] Lustre: server umount lustre-OST0000 complete [ 2809.125339] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 2812.425600] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2815.872943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2818.159328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2820.257157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2822.471519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2825.287796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2825.336326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.470039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2825.488761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2825.530906] Lustre: lustre-MDT0000: new disk, initializing [ 2825.563779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.571761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2827.063270] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2832.013719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2832.045113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2832.097640] Lustre: 176676:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2832.134506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2832.136102] Lustre: Skipped 1 previous similar message [ 2832.175796] Lustre: lustre-MDT0001: new disk, initializing [ 2832.221462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2832.236420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2832.240677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2833.581118] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2836.657904] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2837.966592] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2840.252883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2840.275923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2840.395087] Lustre: lustre-OST0000: new disk, initializing [ 2840.396885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2840.430476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.286406] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2842.999240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2843.003591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2843.033434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2845.387041] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2846.839731] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2846.918370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2848.140788] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2848.207785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2853.348750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2853.349606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.352130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2853.352134] Lustre: Skipped 2 previous similar messages [ 2853.352780] LustreError: Skipped 1 previous similar message [ 2853.355583] Lustre: Skipped 2 previous similar messages [ 2858.466706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.468796] Lustre: Skipped 2 previous similar messages [ 2863.072129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2863.087912] LustreError: 179612:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2863.090177] LustreError: 179612:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2863.159406] Lustre: server umount lustre-OST0000 complete [ 2868.193110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2868.197165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.203580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.623213] Lustre: server umount lustre-MDT0000 complete [ 2871.894361] LustreError: 176666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678679 with bad export cookie 2974988041838063152 [ 2871.898476] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2871.968971] LustreError: 180015:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2871.971030] LustreError: 180015:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2872.079331] Lustre: server umount lustre-MDT0001 complete [ 2875.185444] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:18:02 (1756678682) [ 2881.068310] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2882.138227] Key type lgssc unregistered [ 2882.266385] LNet: 181197:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2882.268668] LNetError: 181197:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2882.278305] LNet: Removed LNI 192.168.201.110@tcp [ 2882.570550] Key type .llcrypt unregistered [ 2882.571594] Key type ._llcrypt unregistered [ 2889.676037] Key type ._llcrypt registered [ 2889.677088] Key type .llcrypt registered [ 2889.720578] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2890.299835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2890.311470] alg: No test for adler32 (adler32-zlib) [ 2891.186836] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2891.291793] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2892.880166] Key type lgssc registered [ 2893.199908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2895.754187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2895.761260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2896.942241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.203535] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2901.137686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2901.261309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2902.547967] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2904.335848] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2905.666855] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2908.215210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2908.309564] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2908.311615] LustreError: 184010:0:(llog.c:1642:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2908.316166] Lustre: 184010:0:(mgc_request_server.c:703:mgc_llog_local_copy()) MGC192.168.201.110@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2908.360308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.274815] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2912.023798] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2912.767898] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.768546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2912.774335] Lustre: Skipped 1 previous similar message [ 2912.776842] Lustre: Skipped 2 previous similar messages [ 2917.858664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.860643] Lustre: Skipped 2 previous similar messages [ 2922.978586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2922.980148] Lustre: Skipped 2 previous similar messages [ 2927.072196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2927.108957] LustreError: 184662:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2927.135124] Lustre: server umount lustre-MDT0000 complete [ 2928.097948] LustreError: 182373:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.102594] LustreError: 182373:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 2928.314164] LustreError: 182353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678735 with bad export cookie 8016079369958009017 [ 2928.317078] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2928.318429] LustreError: 182353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2928.387205] LustreError: 184867:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2928.389016] LustreError: 184867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2928.478620] Lustre: server umount lustre-MDT0001 complete [ 2939.954182] LustreError: 185068:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2939.957482] LustreError: 185068:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2940.042682] Lustre: server umount lustre-OST0000 complete [ 2943.026603] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 2946.410739] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2949.913114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2952.261483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2954.377892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2956.525285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2958.812060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2958.831302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.920431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2958.931796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2958.969628] Lustre: lustre-MDT0000: new disk, initializing [ 2958.994179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.000647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2960.381732] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2964.623905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2964.648994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2964.682995] Lustre: 187404:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2964.705866] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2964.708029] Lustre: Skipped 1 previous similar message [ 2964.710142] LustreError: 188168:0:(llog.c:1642:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2964.713914] LustreError: 188168:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 2964.715691] Lustre: 188168:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2964.725905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2964.727634] Lustre: Skipped 1 previous similar message [ 2964.764614] Lustre: lustre-MDT0001: new disk, initializing [ 2964.801098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2964.815051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2964.819796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2966.142432] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2969.234388] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2970.488025] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2972.709269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2972.734511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2972.844595] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2972.846030] Lustre: Skipped 1 previous similar message [ 2972.847973] LustreError: 189124:0:(llog.c:1642:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2972.850732] LustreError: 189124:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 2972.852894] Lustre: 189124:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2972.855962] Lustre: 189124:0:(mgc_request_server.c:726:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2972.864804] Lustre: lustre-OST0000: new disk, initializing [ 2972.866668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2972.896812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.750811] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2977.752732] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2979.071165] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2982.903842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2982.908379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2982.937540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2983.235044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2984.485270] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2984.553741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2988.002393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2988.003993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.005069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2988.008847] Lustre: Skipped 2 previous similar messages [ 2988.011440] Lustre: Skipped 1 previous similar message [ 2991.213936] LustreError: 190372:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2991.215916] LustreError: 190372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2991.291530] Lustre: server umount lustre-OST0000 complete [ 2995.681142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2995.683444] LustreError: Skipped 1 previous similar message [ 2995.685709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.690501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.693240] Lustre: Skipped 2 previous similar messages [ 2998.590950] LustreError: 190573:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2998.592732] LustreError: 190573:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2998.620887] Lustre: server umount lustre-MDT0000 complete [ 2999.814845] LustreError: 187395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678807 with bad export cookie 8016079369958009703 [ 2999.818631] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2999.820482] LustreError: 187395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2999.986845] Lustre: server umount lustre-MDT0001 complete [ 3002.624139] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:20:09 (1756678809) [ 3007.956867] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3011.661592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.822135] LustreError: 192139:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.836979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.184505] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3016.059765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3017.442298] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3018.405572] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3020.539317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3022.538430] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3025.306098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3025.333081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3025.390715] Lustre: lustre-OST0001: new disk, initializing [ 3025.392535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3027.316673] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3027.457636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3027.462498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3027.497789] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3032.776554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3039.566248] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3045.184854] Lustre: DEBUG MARKER: ADD OST3 [ 3046.882402] loop: module loaded [ 3046.902698] loop0: detected capacity change from 0 to 204800000 [ 3046.977084] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3046.982930] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3046.988475] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3047.040603] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3049.105576] loop0: detected capacity change from 0 to 204800000 [ 3051.029498] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3051.049450] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3051.122257] Lustre: lustre-OST0003: new disk, initializing [ 3051.124991] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3051.175265] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3051.177898] Lustre: Skipped 3 previous similar messages [ 3052.033128] Lustre: DEBUG MARKER: STOP OST3 [ 3052.521292] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3052.556945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3052.561684] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3052.571956] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3052.763526] Lustre: Failing over lustre-OST0003 [ 3052.785789] LustreError: 196478:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3052.789238] LustreError: 196478:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3052.807943] Lustre: server umount lustre-OST0003 complete [ 3053.430964] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3054.560074] LustreError: 196412:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3056.212540] LustreError: 193594:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.292747] Lustre: DEBUG MARKER: START OST3 again [ 3057.634058] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3057.637887] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.643808] Lustre: Skipped 1 previous similar message [ 3059.561206] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3059.645719] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3061.168668] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3061.407104] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3061.407263] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3061.521021] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3075.062877] Lustre: Failing over lustre-OST0003 [ 3075.089100] LustreError: 198429:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3075.090855] LustreError: 198429:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3075.106912] Lustre: server umount lustre-OST0003 complete [ 3077.090130] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3077.090552] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.092401] LustreError: Skipped 1 previous similar message [ 3077.095698] Lustre: Skipped 1 previous similar message [ 3077.096468] LustreError: 193775:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.102534] LustreError: 193775:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 3085.283676] LustreError: 198173:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.288332] LustreError: 198173:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3087.841563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3087.844103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.847711] Lustre: Skipped 1 previous similar message [ 3087.849433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3087.852446] Lustre: Skipped 2 previous similar messages [ 3093.784114] Lustre: server umount lustre-MDT0000 complete [ 3095.105072] LustreError: 192857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678902 with bad export cookie 8016079369958010571 [ 3095.111758] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3095.114444] LustreError: 192857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3095.521776] LustreError: 194577:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.525441] LustreError: 194577:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3101.434412] Lustre: server umount lustre-MDT0001 complete [ 3102.746921] Lustre: server umount lustre-OST0000 complete [ 3104.029552] Lustre: server umount lustre-OST0001 complete [ 3106.232442] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3109.069861] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3112.975165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3115.605313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3118.640072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3121.749384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3124.981410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3125.009316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.114087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3125.127262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3125.167490] Lustre: lustre-MDT0000: new disk, initializing [ 3125.199882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3125.201520] Lustre: Skipped 1 previous similar message [ 3125.207720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3126.883496] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3132.248045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.272638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3132.305293] Lustre: 201486:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3132.428092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3133.927524] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3137.607413] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3138.984807] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3142.338617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3142.406854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3142.600191] Lustre: lustre-OST0000: new disk, initializing [ 3142.603132] Lustre: Skipped 1 previous similar message [ 3142.606608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3142.610382] Lustre: Skipped 2 previous similar messages [ 3143.828989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3143.835146] Lustre: Skipped 1 previous similar message [ 3143.839305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3143.892361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3145.219398] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3149.196587] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3151.021315] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3151.111297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3152.899289] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3152.979554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3154.407577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.408536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.412168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3154.412178] Lustre: Skipped 8 previous similar messages [ 3154.412364] LustreError: Skipped 1 previous similar message [ 3154.420311] Lustre: Skipped 6 previous similar messages [ 3168.224366] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3168.251136] LustreError: 204424:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3168.254409] LustreError: 204424:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3168.333309] Lustre: server umount lustre-OST0000 complete [ 3175.778222] Lustre: server umount lustre-MDT0000 complete [ 3177.051471] LustreError: 203243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678984 with bad export cookie 8016079369958012825 [ 3177.053979] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.054655] LustreError: 203243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3177.213824] Lustre: server umount lustre-MDT0001 complete [ 3178.415463] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:23:05 (1756678985) [ 3183.435508] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3186.248848] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3189.535809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3191.618824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3193.582633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3195.742603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3198.084413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3198.109590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.196628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3198.212371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3198.251138] Lustre: lustre-MDT0000: new disk, initializing [ 3198.284702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3199.719192] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3203.936921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3203.960227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3203.991559] Lustre: 207666:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3204.097091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3205.398435] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3208.447114] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3209.668705] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3211.913810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3211.937145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3213.166099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3213.916557] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3216.923461] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3218.226804] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3218.291262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3219.486347] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3219.554646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3222.285519] Lustre: server umount lustre-OST0000 complete [ 3224.545349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3224.547465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.550862] Lustre: Skipped 2 previous similar messages [ 3224.552622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3224.554101] Lustre: Skipped 9 previous similar messages [ 3233.765384] LustreError: 210804:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3233.767455] LustreError: 210804:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3233.792775] Lustre: server umount lustre-MDT0000 complete [ 3234.898693] LustreError: 207657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679042 with bad export cookie 8016079369958013707 [ 3234.901284] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3234.901736] LustreError: 207657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3244.612030] Lustre: server umount lustre-MDT0001 complete [ 3247.527642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.681671] LustreError: 211482:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.686578] LustreError: 211482:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3248.871675] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3251.826041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3253.298022] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3255.008410] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3256.075373] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3258.498495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3258.634654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.636406] Lustre: Skipped 7 previous similar messages [ 3260.734401] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3262.897588] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3282.401426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3286.543877] Lustre: server umount lustre-OST0000 complete [ 3293.153901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.160825] Lustre: Skipped 4 previous similar messages [ 3293.164473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.167793] Lustre: Skipped 8 previous similar messages [ 3294.577692] Lustre: server umount lustre-MDT0000 complete [ 3296.120345] LustreError: 211463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679103 with bad export cookie 8016079369958014589 [ 3296.128069] LustreError: 211463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3296.128679] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3296.313268] Lustre: server umount lustre-MDT0001 complete [ 3398.905699] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 3400.081400] Key type lgssc unregistered [ 3400.224513] LNet: 214815:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3400.228550] LNetError: 214815:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3400.236396] LNet: Removed LNI 192.168.201.110@tcp [ 3400.589126] Key type .llcrypt unregistered [ 3400.590331] Key type ._llcrypt unregistered [ 3409.712321] Key type ._llcrypt registered [ 3409.713379] Key type .llcrypt registered [ 3409.765470] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3416.743442] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3417.213539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3417.310655] alg: No test for adler32 (adler32-zlib) [ 3418.234325] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3418.355318] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 3419.944303] Key type lgssc registered [ 3420.578059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3424.573533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3427.083441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3429.821785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3432.598977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3435.343489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3435.386675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3435.399106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.502142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3436.519906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3436.573029] Lustre: lustre-MDT0000: new disk, initializing [ 3436.614328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.626760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3438.471752] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3443.425252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3443.448872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3443.479305] Lustre: 218124:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3443.503712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3443.505302] Lustre: Skipped 1 previous similar message [ 3443.538934] Lustre: lustre-MDT0001: new disk, initializing [ 3443.577436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3443.590772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3443.593911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3445.009947] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3448.415012] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3449.688428] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3452.365158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3452.391075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3452.498864] Lustre: lustre-OST0000: new disk, initializing [ 3452.500718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3452.531040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.491812] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3455.364683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3455.369873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3455.420043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3457.732370] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3459.190697] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3459.276539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3460.704952] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3460.801742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3463.533430] LustreError: 221062:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3463.605690] Lustre: server umount lustre-OST0000 complete [ 3465.700331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.701065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.704226] Lustre: Skipped 1 previous similar message [ 3470.818249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3470.820655] Lustre: Skipped 2 previous similar messages [ 3474.917775] LustreError: 221263:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3474.921109] LustreError: 221263:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3474.959743] Lustre: server umount lustre-MDT0000 complete [ 3476.463591] LustreError: 219868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679284 with bad export cookie 4611328075025667045 [ 3476.467594] LustreError: 219868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3476.468886] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.694960] LustreError: 221465:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3486.697580] LustreError: 221465:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3486.836993] Lustre: server umount lustre-MDT0001 complete [ 3493.296096] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3497.518690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.723153] LustreError: 222638:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.732955] LustreError: 222638:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3497.748696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.274791] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3503.044720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3503.074233] LustreError: 222638:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3503.206699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3504.596810] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3505.592111] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3508.391108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3510.568475] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3513.701030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3513.735461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3513.807532] Lustre: lustre-OST0001: new disk, initializing [ 3513.809384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3513.852873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3513.854731] Lustre: Skipped 1 previous similar message [ 3515.795371] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3520.036395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3520.043741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3520.104878] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3526.081453] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3535.228321] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3544.522799] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:29:11 (1756679351) [ 3549.155658] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3549.156991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3549.157805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3549.158653] Lustre: Skipped 3 previous similar messages [ 3551.380417] LustreError: 226014:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3551.384712] LustreError: 226014:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3551.426782] Lustre: server umount lustre-MDT0000 complete [ 3553.076679] LustreError: 223739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679360 with bad export cookie 4611328075025667913 [ 3553.083726] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3553.263099] Lustre: server umount lustre-MDT0001 complete [ 3564.922254] LustreError: 226418:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3564.926147] LustreError: 226418:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3564.953828] Lustre: server umount lustre-OST0000 complete [ 3576.828294] LustreError: 226620:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3576.830597] LustreError: 226620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3576.941877] Lustre: server umount lustre-OST0001 complete [ 3579.535545] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3583.223418] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3587.080035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3589.538214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3591.879642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3594.790615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3598.129852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3598.165053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.281797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3598.300432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3598.340251] Lustre: lustre-MDT0000: new disk, initializing [ 3598.366373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.374625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3599.723373] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3604.283606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3604.305029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3604.346256] Lustre: 228819:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3604.467469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3605.836953] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3609.416936] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3610.744417] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3613.448231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3613.483254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3613.650750] Lustre: lustre-OST0000: new disk, initializing [ 3613.654889] Lustre: Skipped 1 previous similar message [ 3613.659934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3613.663648] Lustre: Skipped 2 previous similar messages [ 3615.104277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3615.106441] Lustre: Skipped 1 previous similar message [ 3615.108717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3615.156638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3615.790502] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3619.360283] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3620.840680] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3620.936387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3622.865659] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3622.992996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3625.444384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3625.446453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.447803] LustreError: Skipped 1 previous similar message [ 3625.451510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3625.454568] Lustre: Skipped 4 previous similar messages [ 3630.561921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3630.563471] Lustre: Skipped 1 previous similar message [ 3638.240193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3638.262153] LustreError: 231759:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3638.264050] LustreError: 231759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3638.348289] Lustre: server umount lustre-OST0000 complete [ 3640.289459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3640.292159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.296070] Lustre: Skipped 1 previous similar message [ 3640.298361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3640.300636] Lustre: Skipped 2 previous similar messages [ 3646.065520] Lustre: server umount lustre-MDT0000 complete [ 3647.825175] LustreError: 228811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679455 with bad export cookie 4611328075025670041 [ 3647.831717] LustreError: 228811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3647.832193] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3648.052107] Lustre: server umount lustre-MDT0001 complete [ 3651.545054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.699461] LustreError: 232640:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.725151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3651.726848] Lustre: Skipped 2 previous similar messages [ 3653.267894] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3656.606870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3658.321886] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3660.488126] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3662.058241] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3664.778618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3667.299241] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3669.396658] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3815.393959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3815.396288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.399695] Lustre: Skipped 1 previous similar message [ 3815.401309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3815.402953] Lustre: Skipped 3 previous similar messages [ 3819.695087] LustreError: 234928:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3819.696730] LustreError: 234928:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3819.763423] Lustre: server umount lustre-OST0000 complete [ 3827.287064] Lustre: server umount lustre-MDT0000 complete [ 3828.679322] LustreError: 232620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679636 with bad export cookie 4611328075025670923 [ 3828.683451] LustreError: 232620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3828.683784] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.858448] Lustre: server umount lustre-MDT0001 complete [ 3935.736756] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 3937.070931] Key type lgssc unregistered [ 3937.199406] LNet: 235898:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3937.201701] LNetError: 235898:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3937.209301] LNet: Removed LNI 192.168.201.110@tcp [ 3937.509126] Key type .llcrypt unregistered [ 3937.510295] Key type ._llcrypt unregistered [ 3946.821410] Key type ._llcrypt registered [ 3946.822390] Key type .llcrypt registered [ 3946.865071] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3953.196932] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3953.636486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3953.667784] alg: No test for adler32 (adler32-zlib) [ 3954.510831] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3954.586790] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 3956.160147] Key type lgssc registered [ 3956.717779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3959.799433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3962.481518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3964.577814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3966.975638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3972.733753] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3976.822500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3976.841757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3976.848081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.936541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3977.949256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3977.990046] Lustre: lustre-MDT0000: new disk, initializing [ 3978.014355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.020561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3979.297391] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3983.678695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3983.722656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3983.775288] Lustre: 239897:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3983.832608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3983.835551] Lustre: Skipped 1 previous similar message [ 3983.875446] Lustre: lustre-MDT0001: new disk, initializing [ 3983.918106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3983.934259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3983.937677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3985.285315] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3987.855473] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3990.343572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3990.368257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3990.467115] Lustre: lustre-OST0000: new disk, initializing [ 3990.468788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3990.498565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.984238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3990.988671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3991.045332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3992.815354] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3997.689656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3997.713104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3997.785371] Lustre: lustre-OST0001: new disk, initializing [ 3997.787486] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3997.833046] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3999.976779] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4005.886824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4006.918566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4006.925254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4006.967579] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4008.751496] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4019.680836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4019.682937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.686560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.194890] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.195758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.200019] Lustre: Skipped 1 previous similar message [ 4020.205734] Lustre: Skipped 3 previous similar messages [ 4023.173812] LustreError: 243424:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4023.204779] Lustre: server umount lustre-MDT0000 complete [ 4024.557576] LustreError: 239888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679832 with bad export cookie 3522498387876952848 [ 4024.562132] LustreError: 239888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4024.562691] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4024.632446] LustreError: 243626:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4024.635016] LustreError: 243626:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4024.722781] Lustre: server umount lustre-MDT0001 complete [ 4036.279490] LustreError: 243826:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4036.283053] LustreError: 243826:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4036.302760] Lustre: server umount lustre-OST0000 complete [ 4047.867230] LustreError: 244028:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4047.870974] LustreError: 244028:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4047.983758] Lustre: server umount lustre-OST0001 complete [ 4053.380657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4057.411054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4057.759619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4061.175182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4064.689515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4065.105455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4072.799475] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4077.066894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.080721] Lustre: lustre-MDT0000: reset Object Index mappings [ 4077.302185] LustreError: 247336:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.310757] LustreError: 247336:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4077.329491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.889530] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4082.242302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4082.255497] Lustre: lustre-MDT0001: reset Object Index mappings [ 4084.039464] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4085.336760] Lustre: 248447:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4087.993351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4088.132410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.134538] Lustre: Skipped 1 previous similar message [ 4090.556300] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4092.262804] LustreError: 248802:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.541625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4096.145709] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4101.933320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4101.934313] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4104.507435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4105.690992] Lustre: 250291:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4112.354855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.358352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4112.358742] Lustre: Skipped 3 previous similar messages [ 4112.361724] Lustre: Skipped 1 previous similar message [ 4117.473173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4117.476253] Lustre: Skipped 3 previous similar messages [ 4118.094437] LustreError: 250397:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4118.097653] LustreError: 250397:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4118.137356] Lustre: server umount lustre-MDT0000 complete [ 4119.639154] LustreError: 250293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679927 with bad export cookie 3522498387876959617 [ 4119.646232] LustreError: 250293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4119.647125] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.848924] Lustre: server umount lustre-MDT0001 complete [ 4130.814144] LustreError: 250799:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4130.818198] LustreError: 250799:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4130.843092] Lustre: server umount lustre-OST0000 complete [ 4141.730974] Lustre: server umount lustre-OST0001 complete [ 4148.934486] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:39:15 (1756679955) [ 4154.616523] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4158.042499] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4161.872666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4164.531466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.544518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.100201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4173.150371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4173.196902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4173.310105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4173.328627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4173.381338] Lustre: lustre-MDT0000: new disk, initializing [ 4173.411276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.413948] Lustre: Skipped 1 previous similar message [ 4173.421181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4174.797093] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4179.098152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4179.119048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4179.148672] Lustre: 254516:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4179.257325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4180.639740] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4183.774045] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4185.391858] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4188.247335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4188.275647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4188.417483] Lustre: lustre-OST0000: new disk, initializing [ 4188.420504] Lustre: Skipped 1 previous similar message [ 4188.423398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4188.426062] Lustre: Skipped 2 previous similar messages [ 4189.701240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4189.703900] Lustre: Skipped 1 previous similar message [ 4189.706398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4189.750133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4190.540797] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4193.745342] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4195.475140] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4195.578614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4197.044980] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4197.131205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4199.909886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4199.912131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.913457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4199.916494] Lustre: Skipped 2 previous similar messages [ 4199.919956] Lustre: Skipped 1 previous similar message [ 4210.150093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4210.153503] Lustre: Skipped 3 previous similar messages [ 4212.192202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4212.218171] LustreError: 257453:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4212.221850] LustreError: 257453:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4212.334779] Lustre: server umount lustre-OST0000 complete [ 4215.267721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.267745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4215.275453] Lustre: Skipped 1 previous similar message [ 4215.281581] LustreError: Skipped 1 previous similar message [ 4219.734609] Lustre: server umount lustre-MDT0000 complete [ 4220.385314] LustreError: 254526:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.390875] LustreError: 254526:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4220.937335] LustreError: 254507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680028 with bad export cookie 3522498387876961710 [ 4220.941099] LustreError: 254507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4220.941271] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.100370] Lustre: server umount lustre-MDT0001 complete [ 4228.242286] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4232.464930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.616899] LustreError: 259031:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4232.631899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.633595] Lustre: Skipped 2 previous similar messages [ 4234.039670] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4237.432862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4238.871138] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4239.904674] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4242.529512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4244.795240] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4248.073736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4248.104665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4248.165713] Lustre: lustre-OST0001: new disk, initializing [ 4248.167438] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4249.804129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4249.807745] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4249.848584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4250.408421] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4255.515476] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4256.916685] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4263.601867] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4263.605396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4268.513583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4268.517754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.525443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4268.528690] Lustre: Skipped 2 previous similar messages [ 4272.143812] LustreError: 262538:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4272.147554] LustreError: 262538:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4272.185696] Lustre: server umount lustre-MDT0000 complete [ 4273.540501] LustreError: 259056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680081 with bad export cookie 3522498387876962592 [ 4273.543850] LustreError: 259056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4273.545856] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4273.715474] Lustre: server umount lustre-MDT0001 complete [ 4284.886499] Lustre: server umount lustre-OST0000 complete [ 4295.574157] Lustre: server umount lustre-OST0001 complete [ 4301.184855] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4305.471694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.646388] LustreError: 264313:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.672149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.673858] Lustre: Skipped 3 previous similar messages [ 4306.983150] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4310.280975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4311.796627] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4312.714921] Lustre: 265425:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4315.179301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4316.390288] LustreError: 265779:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.402836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4317.703570] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4321.331219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.868324] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4323.657615] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4326.766669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4327.986563] Lustre: 267275:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4334.899617] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4334.904456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4338.357764] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:42:25 (1756680145) [ 4350.440783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4350.445889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4350.447631] Lustre: Skipped 5 previous similar messages [ 4350.452995] Lustre: Skipped 7 previous similar messages [ 4356.416177] LustreError: 267985:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4356.417819] LustreError: 267985:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4356.452797] Lustre: server umount lustre-MDT0000 complete [ 4357.923141] LustreError: 266599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680165 with bad export cookie 3522498387876968878 [ 4357.926493] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4357.928793] LustreError: 266599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4360.673364] LustreError: 265932:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.678231] LustreError: 265932:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4364.158931] Lustre: server umount lustre-MDT0001 complete [ 4380.128177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4380.180782] Lustre: server umount lustre-OST0000 complete [ 4388.081257] Lustre: server umount lustre-OST0001 complete [ 4390.307525] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4394.263307] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4398.362949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4401.076365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4403.924756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4406.528746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4414.029099] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4418.477037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4418.509391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.601338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4418.614584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4418.659788] Lustre: lustre-MDT0000: new disk, initializing [ 4418.690663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4420.275853] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4425.624867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4425.657613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4425.698213] Lustre: 271471:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4425.807632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4427.442463] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4432.625095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4432.662987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4433.868724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4435.345187] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4440.691587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4440.733323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4440.885746] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4440.889027] Lustre: Skipped 6 previous similar messages [ 4442.197576] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4443.303674] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4448.417276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4459.539890] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 18:44:26 (1756680266) [ 4461.536817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4461.539622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.543369] Lustre: Skipped 3 previous similar messages [ 4461.545988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.547863] Lustre: Skipped 8 previous similar messages [ 4466.461472] Lustre: server umount lustre-MDT0000 complete [ 4467.644332] LustreError: 273071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680275 with bad export cookie 3522498387876976732 [ 4467.648278] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4467.649303] LustreError: 273071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4467.681608] LustreError: 271477:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.687374] LustreError: 271477:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4467.805060] Lustre: server umount lustre-MDT0001 complete [ 4475.344232] Lustre: server umount lustre-OST0000 complete [ 4477.009462] Lustre: server umount lustre-OST0001 complete [ 4483.504546] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4487.313961] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4490.769863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.210721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.429148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.099079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4500.811134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4500.833278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.906375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4500.909196] Lustre: Skipped 2 previous similar messages [ 4500.918568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4500.920138] Lustre: Skipped 4 previous similar messages [ 4500.953411] Lustre: lustre-MDT0000: new disk, initializing [ 4500.954612] Lustre: Skipped 3 previous similar messages [ 4500.982310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4500.984745] Lustre: Skipped 3 previous similar messages [ 4502.245457] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4506.953744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4506.991850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4507.045615] Lustre: 278521:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4507.183517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4507.189570] Lustre: Skipped 2 previous similar messages [ 4508.681750] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4512.022969] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4513.547483] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4516.798610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4516.829688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4518.413705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4519.525661] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4522.996284] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4523.928711] Lustre: Mounted lustre-client [ 4524.253936] LustreError: 281027:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9979903d2800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4524.258201] LustreError: 281027:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4524.259847] LustreError: 281027:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 4524.277975] Lustre: Unmounted lustre-client [ 4528.609401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4528.609420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.612055] LustreError: Skipped 1 previous similar message [ 4528.616238] Lustre: Skipped 5 previous similar messages [ 4533.871812] Lustre: server umount lustre-OST0000 complete [ 4541.694355] Lustre: server umount lustre-MDT0000 complete [ 4542.910239] LustreError: 278514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680350 with bad export cookie 3522498387876978335 [ 4542.913106] LustreError: 278514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4542.913505] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4543.043663] Lustre: server umount lustre-MDT0001 complete [ 4547.147420] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4548.062104] Key type lgssc unregistered [ 4585.056361] LNet: 282303:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4585.058803] LNetError: 282303:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4654.694518] LNet: Removed LNI 192.168.201.110@tcp [ 4655.069111] Key type .llcrypt unregistered [ 4655.070129] Key type ._llcrypt unregistered [ 4661.318811] Key type ._llcrypt registered [ 4661.319805] Key type .llcrypt registered [ 4661.357404] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4668.368417] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4668.772565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4668.783083] alg: No test for adler32 (adler32-zlib) [ 4669.623185] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4669.699857] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 4671.272148] Key type lgssc registered [ 4671.656602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4674.846497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4677.191177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4679.331833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4681.745187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4682.442216] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 18:48:09 (1756680489) [ 4701.866638] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 18:48:28 (1756680508) [ 4716.577317] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4717.643260] Key type lgssc unregistered [ 4717.778384] LNet: 288021:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4717.782278] LNetError: 288021:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4717.792525] LNet: Removed LNI 192.168.201.110@tcp [ 4718.194531] Key type .llcrypt unregistered [ 4718.196057] Key type ._llcrypt unregistered [ 4751.881231] Key type ._llcrypt registered [ 4751.882795] Key type .llcrypt registered [ 4751.922764] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4752.045264] Key type .llcrypt unregistered [ 4752.046700] Key type ._llcrypt unregistered [ 4763.999897] Key type ._llcrypt registered [ 4764.001183] Key type .llcrypt registered [ 4764.035254] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4764.149376] Key type .llcrypt unregistered [ 4764.151240] Key type ._llcrypt unregistered [ 4796.400064] Key type ._llcrypt registered [ 4796.401080] Key type .llcrypt registered [ 4796.441808] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4796.544371] Key type .llcrypt unregistered [ 4796.545392] Key type ._llcrypt unregistered [ 4806.781325] Key type ._llcrypt registered [ 4806.782489] Key type .llcrypt registered [ 4806.814044] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4806.902280] Key type .llcrypt unregistered [ 4806.903365] Key type ._llcrypt unregistered [ 4820.049106] Key type ._llcrypt registered [ 4820.050168] Key type .llcrypt registered [ 4820.086237] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4820.186189] Key type .llcrypt unregistered [ 4820.187214] Key type ._llcrypt unregistered [ 4850.451176] Key type ._llcrypt registered [ 4850.452430] Key type .llcrypt registered [ 4850.491898] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4850.615171] Key type .llcrypt unregistered [ 4850.616429] Key type ._llcrypt unregistered [ 4866.328000] Key type ._llcrypt registered [ 4866.329993] Key type .llcrypt registered [ 4866.400079] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4866.577502] Key type .llcrypt unregistered [ 4866.580191] Key type ._llcrypt unregistered [ 4881.706334] Key type ._llcrypt registered [ 4881.708149] Key type .llcrypt registered [ 4881.776181] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4881.873288] Key type .llcrypt unregistered [ 4881.874366] Key type ._llcrypt unregistered [ 4901.794729] Key type ._llcrypt registered [ 4901.795690] Key type .llcrypt registered [ 4901.830480] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4907.843143] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4908.304323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4908.325532] alg: No test for adler32 (adler32-zlib) [ 4909.177438] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4909.259572] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 4910.840089] Key type lgssc registered [ 4911.189688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4914.637438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4917.439614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4919.550956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.768134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4922.387297] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4791 sec ========== 18:52:09 (1756680729) [ 4922.965603] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:52:09 (1756680729) === [ 4924.163706] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:52:11 (1756680731) === [ 4937.078488] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4938.130650] Key type lgssc unregistered [ 4938.273440] LNet: 303891:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4938.275513] LNetError: 303891:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4938.284371] LNet: Removed LNI 192.168.201.110@tcp [ 4938.612750] Key type .llcrypt unregistered [ 4938.613745] Key type ._llcrypt unregistered