[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494939240 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002337] x2apic enabled [ 0.003022] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007472] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010104] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.022099] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027571] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] *** VALIDATE tmpfs *** [ 0.031136] *** VALIDATE proc *** [ 0.033293] *** VALIDATE cgroup *** [ 0.034000] *** VALIDATE cgroup2 *** [ 0.034255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036004] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042466] debug: unmapping init [mem 0xffffffff95c59000-0xffffffff95c60fff] [ 0.045168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054315] rcu: Hierarchical SRCU implementation. [ 0.056504] smp: Bringing up secondary CPUs ... [ 0.057592] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.062171] smp: Brought up 1 node, 4 CPUs [ 0.064015] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147595] node 0 deferred pages initialised in 80ms [ 0.151016] devtmpfs: initialized [ 0.152311] x86/mm: Memory block size: 128MB [ 0.154775] gcov: version magic: 0x41383552 [ 0.155606] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160286] pinctrl core: initialized pinctrl subsystem [ 0.162145] [ 0.162704] ************************************************************* [ 0.165012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167010] ** ** [ 0.169013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171014] ** ** [ 0.173012] ** This means that this kernel is built to expose internal ** [ 0.175010] ** IOMMU data structures, which may compromise security on ** [ 0.176008] ** your system. ** [ 0.178012] ** ** [ 0.181011] ** If you see this message and you are not debugging the ** [ 0.182010] ** kernel, report this immediately to your vendor! ** [ 0.183012] ** ** [ 0.186015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188009] ************************************************************* [ 0.189696] NET: Registered protocol family 16 [ 0.190000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197048] cpuidle: using governor menu [ 0.199356] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201320] PCI: Using configuration type 1 for base access [ 0.205159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218074] cryptd: max_cpu_qlen set to 1000 [ 0.221226] ACPI: Added _OSI(Module Device) [ 0.222021] ACPI: Added _OSI(Processor Device) [ 0.223069] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224029] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229888] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238663] ACPI: Interpreter enabled [ 0.240055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241011] ACPI: Using IOAPIC for interrupt routing [ 0.243094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256786] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271000] acpiphp: Slot [2] registered [ 0.271000] acpiphp: Slot [3] registered [ 0.273082] acpiphp: Slot [4] registered [ 0.274081] acpiphp: Slot [5] registered [ 0.276111] acpiphp: Slot [6] registered [ 0.277000] acpiphp: Slot [7] registered [ 0.277000] acpiphp: Slot [8] registered [ 0.278113] acpiphp: Slot [9] registered [ 0.280112] acpiphp: Slot [10] registered [ 0.281112] acpiphp: Slot [11] registered [ 0.283120] acpiphp: Slot [12] registered [ 0.285091] acpiphp: Slot [13] registered [ 0.286116] acpiphp: Slot [14] registered [ 0.287099] acpiphp: Slot [15] registered [ 0.289082] acpiphp: Slot [16] registered [ 0.290102] acpiphp: Slot [17] registered [ 0.291086] acpiphp: Slot [18] registered [ 0.292103] acpiphp: Slot [19] registered [ 0.293103] acpiphp: Slot [20] registered [ 0.294076] acpiphp: Slot [21] registered [ 0.296103] acpiphp: Slot [22] registered [ 0.297080] acpiphp: Slot [23] registered [ 0.298118] acpiphp: Slot [24] registered [ 0.300122] acpiphp: Slot [25] registered [ 0.301107] acpiphp: Slot [26] registered [ 0.302085] acpiphp: Slot [27] registered [ 0.303072] acpiphp: Slot [28] registered [ 0.305085] acpiphp: Slot [29] registered [ 0.306076] acpiphp: Slot [30] registered [ 0.307068] acpiphp: Slot [31] registered [ 0.308063] PCI host bridge to bus 0000:00 [ 0.310026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318024] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.320019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325208] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329755] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.342017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.345995] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358454] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361073] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.368916] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.387876] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.392024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399223] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.425020] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.433259] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.441020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.450020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467017] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.478263] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.483014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.495014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.509020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.519793] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.545017] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.560527] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.568016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.595017] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.604295] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.609012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.615017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.627017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.637023] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.640274] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.642315] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.645410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.652166] iommu: Default domain type: Passthrough [ 0.654485] SCSI subsystem initialized [ 0.656167] ACPI: bus type USB registered [ 0.658120] usbcore: registered new interface driver usbfs [ 0.660081] usbcore: registered new interface driver hub [ 0.662079] usbcore: registered new device driver usb [ 0.664161] pps_core: LinuxPPS API ver. 1 registered [ 0.666011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.668041] PTP clock support registered [ 0.669206] EDAC MC: Ver: 3.0.0 [ 0.670530] PCI: Using ACPI for IRQ routing [ 0.671884] NetLabel: Initializing [ 0.672009] NetLabel: domain hash size = 128 [ 0.673009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.674631] NetLabel: unlabeled traffic allowed by default [ 0.675000] vgaarb: loaded [ 0.675239] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.677014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.691086] clocksource: Switched to clocksource kvm-clock [ 0.802630] VFS: Disk quotas dquot_6.6.0 [ 0.804925] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.807950] *** VALIDATE ramfs *** [ 0.809278] *** VALIDATE hugetlbfs *** [ 0.810890] pnp: PnP ACPI init [ 0.813413] pnp: PnP ACPI: found 6 devices [ 0.831657] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.835189] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.837877] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.840041] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.842689] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.844588] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.846888] NET: Registered protocol family 2 [ 0.849023] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.854032] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.856845] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.861195] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.864335] TCP: Hash tables configured (established 65536 bind 65536) [ 0.867526] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.870750] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873126] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.876087] NET: Registered protocol family 1 [ 0.878691] RPC: Registered named UNIX socket transport module. [ 0.881362] RPC: Registered udp transport module. [ 0.883068] RPC: Registered tcp transport module. [ 0.884590] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.886494] NET: Registered protocol family 44 [ 0.887838] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.889882] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.892215] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.894593] PCI: CLS 0 bytes, default 64 [ 0.896238] Unpacking initramfs... [ 2.369775] debug: unmapping init [mem 0xffff9b0bfcc54000-0xffff9b0bfffbffff] [ 2.374371] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.377154] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.380633] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.877085] Initialise system trusted keyrings [ 2.878741] Key type blacklist registered [ 2.880225] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.888909] zbud: loaded [ 2.891730] *** VALIDATE nfs *** [ 2.892787] *** VALIDATE nfs4 *** [ 2.897233] pstore: using deflate compression [ 2.904323] Platform Keyring initialized [ 3.024591] NET: Registered protocol family 38 [ 3.026584] Key type asymmetric registered [ 3.028573] Asymmetric key parser 'x509' registered [ 3.030576] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.034424] io scheduler mq-deadline registered [ 3.036416] io scheduler kyber registered [ 3.038275] io scheduler bfq registered [ 3.040279] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.043336] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.046668] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.049962] ACPI: Power Button [PWRF] [ 3.165723] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.272069] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.454763] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.545295] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.726972] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.759909] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.796915] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.804044] Non-volatile memory driver v1.3 [ 3.805961] Linux agpgart interface v0.103 [ 3.847215] virtio_blk virtio1: [vda] 133856 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.849924] vda: detected capacity change from 0 to 68534272 [ 3.873466] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.875809] vdb: detected capacity change from 0 to 1073741824 [ 3.888486] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.891171] vdc: detected capacity change from 0 to 2621440000 [ 3.906300] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.908658] vdd: detected capacity change from 0 to 2621440000 [ 3.921869] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.924874] vde: detected capacity change from 0 to 4294967296 [ 3.937964] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.940743] vdf: detected capacity change from 0 to 4294967296 [ 3.946558] libphy: Fixed MDIO Bus: probed [ 3.957364] usbcore: registered new interface driver usbserial_generic [ 3.959726] usbserial: USB Serial support registered for generic [ 3.961689] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.965926] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.967463] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.970088] mousedev: PS/2 mouse device common for all mice [ 3.973757] rtc_cmos 00:05: RTC can wake from S4 [ 3.977899] rtc_cmos 00:05: registered as rtc0 [ 3.979982] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.984876] intel_pstate: CPU model not supported [ 3.986506] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.992480] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.995952] hid: raw HID events driver (C) Jiri Kosina [ 3.999897] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.003461] usbcore: registered new interface driver usbhid [ 4.005366] usbhid: USB HID core driver [ 4.006951] drop_monitor: Initializing network drop monitor service [ 4.009164] Initializing XFRM netlink socket [ 4.011054] NET: Registered protocol family 10 [ 4.013810] Segment Routing with IPv6 [ 4.015472] NET: Registered protocol family 17 [ 4.020214] mpls_gso: MPLS GSO support [ 4.025626] RAS: Correctable Errors collector initialized. [ 4.028014] AVX version of gcm_enc/dec engaged. [ 4.030182] AES CTR mode by8 optimization enabled [ 4.103686] sched_clock: Marking stable (4103666719, 0)->(5072247313, -968580594) [ 4.107864] registered taskstats version 1 [ 4.109844] Loading compiled-in X.509 certificates [ 4.112174] zswap: loaded using pool lzo/zbud [ 4.137737] Key type big_key registered [ 4.151632] Key type encrypted registered [ 4.153264] ima: No TPM chip found, activating TPM-bypass! [ 4.155223] ima: Allocated hash algorithm: sha1 [ 4.156750] ima: No architecture policies found [ 4.158147] evm: Initialising EVM extended attributes: [ 4.160351] evm: security.selinux [ 4.161383] evm: security.ima [ 4.162827] evm: security.capability [ 4.164528] evm: HMAC attrs: 0x1 [ 4.168628] rtc_cmos 00:05: setting system clock to 2025-11-18 23:58:47 UTC (1763510327) [ 4.177583] debug: unmapping init [mem 0xffffffff96c03000-0xffffffff96dfffff] [ 4.181089] debug: unmapping init [mem 0xffffffff95982000-0xffffffff95c58fff] [ 4.193122] Write protecting the kernel read-only data: 28672k [ 4.197152] debug: unmapping init [mem 0xffffffff94003000-0xffffffff941fffff] [ 4.199950] debug: unmapping init [mem 0xffffffff94914000-0xffffffff949fffff] [ 4.231950] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.240690] systemd[1]: Detected virtualization kvm. [ 4.242555] systemd[1]: Detected architecture x86-64. [ 4.245152] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.272904] systemd[1]: No hostname configured. [ 4.274802] systemd[1]: Set hostname to . [ 4.276836] random: systemd: uninitialized urandom read (16 bytes read) [ 4.279604] systemd[1]: Initializing machine ID from random generator. [ 4.431214] random: systemd: uninitialized urandom read (16 bytes read) [ 4.433711] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.438182] random: systemd: uninitialized urandom read (16 bytes read) [ 4.440653] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.444639] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.149956] device-mapper: uevent: version 1.0.3 [ 5.152209] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 5.958983] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.024750] scsi host0: ata_piix [ 6.027836] virtio_net virtio0 ens2: renamed from eth0 [ 6.043970] scsi host1: ata_piix [ 6.045272] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.051510] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.327773] dracut-initqueue[593]: RTNETLINK answers: File exists [ 10.670547] random: crng init done [ 10.672076] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.146235] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.412111] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.682486] SELinux: Disabled at runtime. [ 12.747606] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.759719] systemd[1]: Detected virtualization kvm. [ 12.762394] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.238922] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.242117] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.246604] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.250528] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.253823] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.261362] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.268590] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ 13.319272] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 13.676364] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.010984] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.067109] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.175261] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.189707] EDAC sbridge: Ver: 1.1.2 [ 15.774303] Key type dns_resolver registered [ 16.086638] NFS: Registering the id_resolver key type [ 16.088721] Key type id_resolver registered [ 16.090276] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 45.146148] libcfs: loading out-of-tree module taints kernel. [ 45.164880] Key type ._llcrypt registered [ 45.166203] Key type .llcrypt registered [ 45.279677] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 60.982122] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 62.064183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.074089] alg: No test for adler32 (adler32-zlib) [ 63.595483] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 64.210974] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 65.888161] Key type lgssc registered [ 67.482447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.714029] hrtimer: interrupt took 5655936 ns [ 84.834636] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 85.794885] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.279254] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.702075] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.822160] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.335851] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 125.747925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.790217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 125.809719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.142883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 127.196423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 127.302902] Lustre: lustre-MDT0000: new disk, initializing [ 127.415046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 127.441033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 130.725412] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.377909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.463751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 141.513387] Lustre: 6526:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 141.547669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 141.550683] Lustre: Skipped 1 previous similar message [ 141.642399] Lustre: lustre-MDT0001: new disk, initializing [ 141.725891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 141.758474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 141.775804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 144.635980] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.263631] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 156.582908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.739414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 156.983410] Lustre: lustre-OST0000: new disk, initializing [ 156.996447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 157.084228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.375690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 158.394766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 158.450245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 163.061979] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.377750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.428647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 175.589049] Lustre: lustre-OST0001: new disk, initializing [ 175.598341] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 175.659158] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.556848] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.385216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 185.395088] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 185.436698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 190.676972] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 196.406818] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 206.560243] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 211.172765] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 215.780953] Lustre: DEBUG MARKER: Client: 2.16.61.43 [ 218.354822] Lustre: DEBUG MARKER: MDS: 2.16.61.43 [ 220.899467] Lustre: DEBUG MARKER: OSS: 2.16.61.43 [ 222.382183] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Nov 18 19:02:25 EST 2025 [ 236.742369] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 238.112767] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 254.432740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 254.444811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.484742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 256.995783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 256.997148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 257.018996] Lustre: Skipped 2 previous similar messages [ 257.019920] Lustre: Skipped 2 previous similar messages [ 259.550659] LustreError: 12242:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 259.665704] Lustre: server umount lustre-MDT0000 complete [ 262.411603] LustreError: 6516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510585 with bad export cookie 6314227993301339525 [ 262.413882] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 262.422591] LustreError: 6516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 267.232803] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 267.234261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.235645] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 267.235652] Lustre: Skipped 1 previous similar message [ 267.243669] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 267.249158] Lustre: Skipped 1 previous similar message [ 268.606910] LustreError: 12443:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 268.613724] LustreError: 12443:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 268.809760] Lustre: server umount lustre-MDT0001 complete [ 271.862757] LustreError: 12644:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 271.871127] LustreError: 12644:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 271.932958] Lustre: server umount lustre-OST0000 complete [ 274.723478] LustreError: 12845:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 274.739598] LustreError: 12845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 274.901524] Lustre: server umount lustre-OST0001 complete [ 280.287722] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 288.053913] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 297.984883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 302.691658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 307.612423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.045608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.971145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 316.021415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 316.175245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 316.198803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 316.246059] Lustre: lustre-MDT0000: new disk, initializing [ 316.319295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 316.333736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 318.765338] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 326.771853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 326.839154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 326.929815] Lustre: 15028:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 326.986546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 326.989844] Lustre: Skipped 1 previous similar message [ 327.050784] Lustre: lustre-MDT0001: new disk, initializing [ 327.152683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 327.202667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 327.217548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 329.950782] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 335.152539] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 338.215089] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 343.894923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.966637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 344.155506] Lustre: lustre-OST0000: new disk, initializing [ 344.157465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 344.234065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.836900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 345.844784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 345.904734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 348.816572] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 354.693851] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 358.042086] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 358.226585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 361.489508] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 361.691645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 366.561572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 366.562800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.565830] LustreError: Skipped 1 previous similar message [ 366.573680] Lustre: Skipped 1 previous similar message [ 366.578340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 366.612037] Lustre: Skipped 2 previous similar messages [ 369.441107] LustreError: 17966:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 369.449036] LustreError: 17966:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 369.597729] Lustre: server umount lustre-OST0000 complete [ 373.217976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 373.225196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.238296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 378.759671] LustreError: 18168:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 378.767857] LustreError: 18168:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 378.924798] Lustre: server umount lustre-MDT0000 complete [ 381.933295] LustreError: 15039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.960296] LustreError: 15039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 382.499462] LustreError: 15020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510705 with bad export cookie 6314227993301341058 [ 382.506171] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.506252] LustreError: 15020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 382.841261] Lustre: server umount lustre-MDT0001 complete [ 388.316126] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:05:11 (1763510711) [ 404.283913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.957071] LustreError: 19506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.075073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.383844] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 410.088397] LustreError: 19506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.109791] LustreError: 19506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 415.208483] LustreError: 19506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.542439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 419.625848] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 424.461255] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 427.591660] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 429.280520] Lustre: Failing over lustre-MDT0000 [ 429.291749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.297261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 429.311636] Lustre: Skipped 2 previous similar messages [ 429.325970] Lustre: Skipped 2 previous similar messages [ 429.423504] LustreError: 20848:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 429.427675] LustreError: 20848:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 429.540516] Lustre: server umount lustre-MDT0000 complete [ 434.658086] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 434.676566] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 445.663187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.728255] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 448.803562] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 451.042820] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 451.045736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 451.068402] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 454.058377] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 461.283252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 461.286619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.282418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.290795] Lustre: Skipped 1 previous similar message [ 461.299669] Lustre: Skipped 1 previous similar message [ 463.195495] LustreError: 21829:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 463.204145] LustreError: 21829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 463.291758] Lustre: server umount lustre-MDT0000 complete [ 466.190474] LustreError: 19487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510789 with bad export cookie 6314227993301342472 [ 466.195552] LustreError: 19487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 466.196829] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 466.488787] Lustre: server umount lustre-MDT0001 complete [ 473.321302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.687210] LustreError: 22507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.698799] LustreError: 22507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 473.750164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.758417] Lustre: Skipped 2 previous similar messages [ 476.919993] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 484.025708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 487.504124] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 491.650667] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 494.498107] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 496.149296] Lustre: Failing over lustre-MDT0000 [ 496.319510] Lustre: server umount lustre-MDT0000 complete [ 499.681582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.683137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.710571] Lustre: Skipped 1 previous similar message [ 508.905769] LustreError: 22503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.916286] LustreError: 22503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 512.701233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.818204] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.052624] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 518.119793] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 518.151554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 518.159459] Lustre: Skipped 1 previous similar message [ 518.180435] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 521.453836] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 528.354907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 528.361612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 528.372091] Lustre: Skipped 3 previous similar messages [ 530.684217] LustreError: 24827:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 530.688571] LustreError: 24827:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 530.796920] Lustre: server umount lustre-MDT0000 complete [ 533.324214] LustreError: 22488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510856 with bad export cookie 6314227993301343403 [ 533.326339] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.333074] LustreError: 22488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 533.597853] Lustre: server umount lustre-MDT0001 complete [ 539.742694] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:07:43 (1763510863) [ 551.747312] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 559.902613] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 568.685751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 574.245519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 578.956519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 583.781198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 590.494487] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 597.231875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 597.272869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.425350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 597.459740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 597.513755] Lustre: lustre-MDT0000: new disk, initializing [ 597.617589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 600.153836] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 607.378476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 607.429725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 607.477317] Lustre: 28497:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 607.642061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 607.651187] Lustre: Skipped 3 previous similar messages [ 607.685916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 610.272603] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 615.642230] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.930670] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 622.382697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 622.430037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 624.313497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 624.363342] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 624.417629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 624.427617] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 626.836461] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 632.436905] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 637.940882] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 655.329852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 655.334346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.337751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.347238] Lustre: Skipped 3 previous similar messages [ 655.354119] Lustre: Skipped 3 previous similar messages [ 656.833069] Lustre: server umount lustre-OST0000 complete [ 666.657053] LustreError: 31356:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 666.659893] LustreError: 31356:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 666.712764] Lustre: server umount lustre-MDT0000 complete [ 669.772984] LustreError: 28489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763510993 with bad export cookie 6314227993301343802 [ 669.779768] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.795989] LustreError: 28489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 670.301176] Lustre: server umount lustre-MDT0001 complete [ 680.521164] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 682.770168] Key type lgssc unregistered [ 683.008855] LNet: 32090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.017402] LNetError: 32090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.026689] LNet: Removed LNI 192.168.203.104@tcp [ 683.804485] Key type .llcrypt unregistered [ 683.806081] Key type ._llcrypt unregistered [ 699.934454] Key type ._llcrypt registered [ 699.936050] Key type .llcrypt registered [ 700.038502] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 700.863810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.967751] alg: No test for adler32 (adler32-zlib) [ 702.028692] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 702.229752] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 703.865436] Key type lgssc registered [ 704.798070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 710.268096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.279517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.639288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 714.425130] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 720.866115] LustreError: 33462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.878118] LustreError: 33462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 722.295752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 722.624599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 726.218551] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 731.546557] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 734.826303] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 740.604894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 740.908224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 745.439803] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 747.092972] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 749.981738] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 765.751507] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:11:29 (1763511089) [ 767.055854] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 768.486623] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 773.696074] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:11:37 (1763511097) [ 779.015266] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:11:42 (1763511102) [ 779.626166] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 785.006977] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 785.012598] Lustre: Skipped 2 previous similar messages [ 786.148374] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:11:49 (1763511109) [ 791.596339] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 791.600988] Lustre: Skipped 4 previous similar messages [ 801.193963] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 801.201508] Lustre: Skipped 5 previous similar messages [ 803.069712] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:12:06 (1763511126) [ 804.898705] Lustre: *** cfs_fail_loc=131b, val=2*** [ 807.120777] Lustre: *** cfs_fail_loc=131b, val=2*** [ 814.025529] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:12:17 (1763511137) [ 818.162520] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 818.170485] Lustre: Skipped 5 previous similar messages [ 826.175280] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:12:29 (1763511149) [ 838.120783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.121396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 838.121732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.127615] Lustre: Skipped 1 previous similar message [ 843.236665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 843.238853] Lustre: Skipped 1 previous similar message [ 848.353752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 848.361323] Lustre: Skipped 1 previous similar message [ 849.376118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 849.413768] LustreError: 40019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 849.498978] Lustre: server umount lustre-OST0000 complete [ 853.473816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.486337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 853.489130] Lustre: Skipped 1 previous similar message [ 858.599346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 866.273943] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 866.351329] LustreError: 40221:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 866.360786] LustreError: 40221:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 866.438511] Lustre: server umount lustre-MDT0000 complete [ 868.834679] LustreError: 33458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.850976] LustreError: 33458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 869.283660] LustreError: 38147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511192 with bad export cookie 3965221975035761407 [ 869.292031] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 869.298954] LustreError: 38147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 869.418250] LustreError: 40424:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 869.422801] LustreError: 40424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 869.604956] Lustre: server umount lustre-MDT0001 complete [ 877.604249] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 880.083870] Key type lgssc unregistered [ 880.335175] LNet: 40955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.344149] LNetError: 40955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.366421] LNet: Removed LNI 192.168.203.104@tcp [ 881.008783] Key type .llcrypt unregistered [ 881.011884] Key type ._llcrypt unregistered [ 893.058024] Key type ._llcrypt registered [ 893.060761] Key type .llcrypt registered [ 893.152661] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 903.457873] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 903.972835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.031156] alg: No test for adler32 (adler32-zlib) [ 904.964098] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 905.103495] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 906.712157] Key type lgssc registered [ 907.487873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 913.429941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 917.649887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 921.610616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 925.927354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 930.481944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 930.539541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 930.557375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.718727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 931.767927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 931.836639] Lustre: lustre-MDT0000: new disk, initializing [ 931.910816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.930773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 934.379629] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 941.344990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 941.390397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 941.446539] Lustre: 44064:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 941.487871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 941.490515] Lustre: Skipped 1 previous similar message [ 941.548117] Lustre: lustre-MDT0001: new disk, initializing [ 941.602261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.621441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 941.625678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 943.784967] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 948.519675] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 950.445990] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 954.579742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 954.626474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 954.802720] Lustre: lustre-OST0000: new disk, initializing [ 954.807541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 954.864659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.184409] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 962.775712] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 964.625052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 964.629804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 964.691079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 965.116178] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 965.211533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 967.503126] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 967.638577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 969.697643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.699990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.710595] LustreError: Skipped 1 previous similar message [ 969.719180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 970.681067] LustreError: 47002:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 970.743802] Lustre: server umount lustre-OST0000 complete [ 974.820159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.822707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.826333] Lustre: Skipped 2 previous similar messages [ 974.834366] Lustre: Skipped 2 previous similar messages [ 978.846979] LustreError: 47203:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 978.855049] LustreError: 47203:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 978.941511] Lustre: server umount lustre-MDT0000 complete [ 981.616996] LustreError: 44054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511304 with bad export cookie 12588016270748326144 [ 981.630334] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 981.740968] LustreError: 47405:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 981.754075] LustreError: 47405:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 981.865064] Lustre: server umount lustre-MDT0001 complete [ 987.783381] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:15:11 (1763511311) [ 992.505081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.972274] LustreError: 48124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.986092] LustreError: 48124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 993.049650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.855200] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 998.372137] LustreError: 48124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.386296] LustreError: 48124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1001.629989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1001.880736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.125714] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1009.820301] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1012.604570] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1017.652969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1017.937693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.707857] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1025.034513] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1032.207080] LustreError: 50324:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1032.884353] Lustre: Modifying parameter general.timeout=1 in log params [ 1034.132946] Lustre: Modifying parameter general.timeout=3 in log params [ 1034.135807] Lustre: Skipped 1 previous similar message [ 1036.137884] Lustre: Modifying parameter general.timeout=6 in log params [ 1036.140098] Lustre: Skipped 2 previous similar messages [ 1040.261283] Lustre: Modifying parameter general.timeout=12 in log params [ 1040.263502] Lustre: Skipped 5 previous similar messages [ 1048.375489] Lustre: Modifying parameter general.timeout=26 in log params [ 1048.378056] Lustre: Skipped 13 previous similar messages [ 1064.482971] Lustre: Modifying parameter general.timeout=63 in log params [ 1064.486237] Lustre: Skipped 36 previous similar messages [ 1099.208503] Lustre: Disabling parameter general.timeout= in log params [ 1099.211140] Lustre: Skipped 66 previous similar messages [ 1101.210467] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:17:04 (1763511424) [ 1102.747942] Lustre: 57237:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1104.643835] Lustre: 57434:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1104.650940] Lustre: 57434:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1106.673252] Lustre: 57646:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1106.679084] Lustre: 57646:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1122.079799] Lustre: 59097:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1122.086024] Lustre: 59097:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1122.094992] LustreError: 59097:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1124.410824] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:17:27 (1763511447) [ 1130.978067] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.979414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.983074] Lustre: Skipped 2 previous similar messages [ 1130.984952] Lustre: Skipped 1 previous similar message [ 1136.097770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.101189] Lustre: Skipped 3 previous similar messages [ 1136.647841] LustreError: 59795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1136.651314] LustreError: 59795:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1136.669090] LustreError: 59795:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000e4f0a15e) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1138.744162] Lustre: server umount lustre-MDT0000 complete [ 1140.509329] LustreError: 48103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511463 with bad export cookie 12588016270748327026 [ 1140.511747] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.514371] LustreError: 48103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1140.652297] Lustre: server umount lustre-MDT0001 complete [ 1152.967191] LustreError: 60200:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1152.971466] LustreError: 60200:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1153.073574] Lustre: server umount lustre-OST0000 complete [ 1161.499064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1164.839171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1167.925585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1171.477483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1176.690321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.965511] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1176.986970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1176.995616] Lustre: Skipped 22 previous similar messages [ 1177.174803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.848540] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1185.983870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1186.078408] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1186.113738] Lustre: 61652:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1186.165898] LustreError: 61664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.069627] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1193.148748] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1196.011436] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1201.432163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1201.579384] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1205.880213] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1210.059875] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1211.391714] Lustre: 63885:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1218.530472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1218.537928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.544327] LustreError: Skipped 1 previous similar message [ 1218.552663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.553746] Lustre: Skipped 1 previous similar message [ 1218.565211] Lustre: Skipped 1 previous similar message [ 1221.875256] LustreError: 64130:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1221.882496] LustreError: 64130:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1221.947087] Lustre: server umount lustre-OST0000 complete [ 1227.237408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1227.241343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.250182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.790625] Lustre: server umount lustre-MDT0000 complete [ 1233.634021] LustreError: 62380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511556 with bad export cookie 12588016270748329616 [ 1233.640802] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.648531] LustreError: 62380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1234.006135] Lustre: server umount lustre-MDT0001 complete [ 1243.153940] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1245.800375] Key type lgssc unregistered [ 1246.063794] LNet: 65072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.067348] LNetError: 65072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.077394] LNet: Removed LNI 192.168.203.104@tcp [ 1246.717585] Key type .llcrypt unregistered [ 1246.720861] Key type ._llcrypt unregistered [ 1263.289057] Key type ._llcrypt registered [ 1263.290631] Key type .llcrypt registered [ 1263.376358] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1263.961793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.994668] alg: No test for adler32 (adler32-zlib) [ 1264.949705] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1265.110704] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1266.752161] Key type lgssc registered [ 1267.729374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1274.257041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1274.281164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.782850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.798467] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1284.977109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1285.403754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1288.139482] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1291.555370] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1294.015285] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1298.526557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1298.892664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.382877] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1305.920105] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1316.367391] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1322.035404] LustreError: 69153:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1322.125321] Lustre: server umount lustre-MDT0000 complete [ 1324.657487] LustreError: 67173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511647 with bad export cookie 8010694421453530804 [ 1324.661761] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1324.664566] LustreError: 67173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1324.688239] LustreError: 66438:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1325.537922] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.540792] LustreError: 66443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.555205] Lustre: Skipped 1 previous similar message [ 1325.578840] LustreError: 66443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1325.587300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1326.843727] LustreError: 69353:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1326.863051] LustreError: 69353:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1327.085107] Lustre: server umount lustre-MDT0001 complete [ 1329.998508] LustreError: 69554:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1330.003855] LustreError: 69554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1330.097620] Lustre: server umount lustre-OST0000 complete [ 1335.715338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1338.403991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1341.145379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1344.206788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1348.733538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.896369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1348.915134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1349.069916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.605406] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1357.636187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1357.714774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1357.744625] Lustre: 71002:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1358.087378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1360.784678] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1364.404175] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1366.648905] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1370.651688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1370.757184] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1370.983777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1374.274307] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1377.358878] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1383.911395] Lustre: 73243:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1385.204994] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1389.032809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.033674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1389.054523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1389.079571] LustreError: Skipped 1 previous similar message [ 1390.199943] LustreError: 73501:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1390.202825] LustreError: 73501:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1390.322537] Lustre: server umount lustre-OST0000 complete [ 1393.634773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1393.640280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.648333] Lustre: Skipped 1 previous similar message [ 1393.651549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.654274] Lustre: Skipped 1 previous similar message [ 1399.130471] LustreError: 73702:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1399.132943] LustreError: 73702:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1399.183194] Lustre: server umount lustre-MDT0000 complete [ 1399.265175] LustreError: 71015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.274910] LustreError: 71015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1401.301646] LustreError: 72646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511724 with bad export cookie 8010694421453531609 [ 1401.303749] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1401.307581] LustreError: 72646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1401.531302] Lustre: server umount lustre-MDT0001 complete [ 1410.051506] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1413.132556] Key type lgssc unregistered [ 1413.489098] LNet: 74436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.499465] LNetError: 74436:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.514333] LNet: Removed LNI 192.168.203.104@tcp [ 1414.257731] Key type .llcrypt unregistered [ 1414.259419] Key type ._llcrypt unregistered [ 1434.250994] Key type ._llcrypt registered [ 1434.252778] Key type .llcrypt registered [ 1434.310947] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1434.950772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1434.969564] alg: No test for adler32 (adler32-zlib) [ 1435.929299] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1436.101207] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1437.744171] Key type lgssc registered [ 1438.532153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.420356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.440635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.747479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.586808] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1454.469147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1454.848950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.353643] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1460.992178] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1463.381332] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1467.824416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1468.201678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.676466] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1476.595885] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1484.403572] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1485.043774] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1486.430835] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1486.433391] Lustre: Skipped 1 previous similar message [ 1488.593757] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1488.609268] Lustre: Skipped 2 previous similar messages [ 1493.286370] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1493.292205] Lustre: Skipped 5 previous similar messages [ 1501.690946] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1501.699302] Lustre: Skipped 12 previous similar messages [ 1517.902613] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1517.905567] Lustre: Skipped 23 previous similar messages [ 1549.903994] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1549.908781] Lustre: Skipped 48 previous similar messages [ 1555.961560] Lustre: 83564:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1557.672344] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:24:41 (1763511881) [ 1558.738512] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1559.908078] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:24:43 (1763511883) [ 1563.619992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1563.628192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.641850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1566.691468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1566.693646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1566.697039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.450098] LustreError: 83859:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1568.522806] Lustre: server umount lustre-OST0000 complete [ 1571.811250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.814794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1571.827955] Lustre: Skipped 1 previous similar message [ 1571.834678] Lustre: Skipped 2 previous similar messages [ 1576.932762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.935733] Lustre: Skipped 1 previous similar message [ 1577.496720] LustreError: 84061:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1577.505616] LustreError: 84061:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1577.583778] Lustre: server umount lustre-MDT0000 complete [ 1580.403405] LustreError: 80386:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763511903 with bad export cookie 8604664496753582157 [ 1580.409292] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.430109] LustreError: 80386:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1580.546437] LustreError: 84262:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1580.556700] LustreError: 84262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1580.675942] Lustre: server umount lustre-MDT0001 complete [ 1587.652656] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1589.616993] Key type lgssc unregistered [ 1589.839377] LNet: 84795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.843336] LNetError: 84795:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.859335] LNet: Removed LNI 192.168.203.104@tcp [ 1590.549456] Key type .llcrypt unregistered [ 1590.550844] Key type ._llcrypt unregistered [ 1593.488407] Key type ._llcrypt registered [ 1593.492245] Key type .llcrypt registered [ 1593.580531] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1597.068354] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules [ 1597.859535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.870109] alg: No test for adler32 (adler32-zlib) [ 1599.002686] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1599.193621] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1600.872454] Key type lgssc registered [ 1601.753987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1602.613964] LustreError: 85959:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1617.135668] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1624.826486] LustreError: 85959:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1625.097568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1625.135053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.589759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.885420] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1635.828398] LustreError: 86965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.853812] LustreError: 86965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1636.503961] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:25:59 (1763511959) [ 1640.297781] LustreError: 86997:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1640.303678] LustreError: 86997:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1640.313533] LustreError: 86997:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 1640.422210] LustreError: 87770:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1640.638167] Lustre: server umount lustre-MDT0000 complete [ 1649.503647] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1651.228304] Key type lgssc unregistered [ 1651.403590] LNet: 88454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.407799] LNetError: 88454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1651.415945] LNet: Removed LNI 192.168.203.104@tcp [ 1651.861348] Key type .llcrypt unregistered [ 1651.863159] Key type ._llcrypt unregistered [ 1664.200657] Key type ._llcrypt registered [ 1664.202473] Key type .llcrypt registered [ 1664.287610] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1664.836141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1664.887780] alg: No test for adler32 (adler32-zlib) [ 1665.850959] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1666.032470] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1667.672179] Key type lgssc registered [ 1668.657862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1673.801922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1673.818824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.235598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.663849] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1683.023050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1683.467949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1686.270382] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1690.450949] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1693.045636] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1697.655040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1698.151981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.795418] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1705.034249] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1742.987465] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:27:46 (1763512066) [ 1743.926235] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1744.879845] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:27:48 (1763512068) [ 1749.475169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1749.482874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1749.484503] Lustre: Skipped 2 previous similar messages [ 1749.495569] Lustre: Skipped 2 previous similar messages [ 1752.802286] LustreError: 92540:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1752.860236] Lustre: server umount lustre-MDT0000 complete [ 1754.600036] LustreError: 90348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1754.618087] LustreError: 90348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1755.142859] LustreError: 89614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512078 with bad export cookie 584390908930215900 [ 1755.152048] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1755.246881] LustreError: 92741:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1755.250513] LustreError: 92741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1755.397680] Lustre: server umount lustre-MDT0001 complete [ 1768.070741] LustreError: 92943:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1768.074567] LustreError: 92943:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1768.361514] Lustre: server umount lustre-OST0000 complete [ 1776.214583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.487210] LustreError: 93573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.609520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.026550] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1781.730868] LustreError: 93573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.222563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1784.655572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1787.112240] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1790.517674] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1792.686695] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1796.210088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1799.796602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1799.835902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1799.933962] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1799.938281] LustreError: 93562:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1799.952073] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1799.957844] LustreError: 95434:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1799.961518] LustreError: 95434:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1799.964397] LustreError: 95434:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1800.010113] Lustre: server umount lustre-OST0000 complete [ 1800.012385] LustreError: 95434:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1805.220349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1805.376041] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1805.380512] LustreError: 93561:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1805.386214] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1805.392705] LustreError: 95896:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1805.396754] LustreError: 95896:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1805.400136] LustreError: 95896:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1805.431523] Lustre: server umount lustre-OST0000 complete [ 1805.433545] LustreError: 95896:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1807.173946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.026680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1811.141487] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1811.145840] Lustre: Found index 0 for lustre-OST0000, updating log [ 1811.149335] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1811.192593] Lustre: lustre-OST0000: new disk, initializing [ 1811.196668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1811.383790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.610750] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1816.756811] Lustre: Failing over lustre-OST0000 [ 1816.780138] LustreError: 96982:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1816.782699] LustreError: 96982:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1816.836493] Lustre: server umount lustre-OST0000 complete [ 1820.130043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1820.138610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.145351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.666674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.667015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.681199] Lustre: Skipped 1 previous similar message [ 1825.256215] LustreError: 97182:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1825.267362] LustreError: 97182:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1825.347480] Lustre: server umount lustre-MDT0000 complete [ 1826.785239] LustreError: 93573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.796067] LustreError: 93573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1827.733572] LustreError: 93553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512151 with bad export cookie 584390908930218637 [ 1827.737121] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.741700] LustreError: 93553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1827.929037] Lustre: server umount lustre-MDT0001 complete [ 1833.270511] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:29:16 (1763512156) [ 1844.089578] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1851.524363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.810151] LustreError: 98752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.886167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.396404] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1861.712656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1866.326348] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1869.048460] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1874.657139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1879.843527] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1886.395506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1886.457604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1886.502283] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1886.576843] Lustre: lustre-OST0001: new disk, initializing [ 1886.581570] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1886.810191] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1886.815067] Lustre: Skipped 2 previous similar messages [ 1889.966523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1889.974145] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1890.057884] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1890.107707] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1894.366677] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1900.221288] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1909.729835] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1909.733856] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.745857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1915.879382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1915.884140] Lustre: Skipped 5 previous similar messages [ 1922.016114] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1922.085607] LustreError: 101991:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1922.088372] LustreError: 101991:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1922.127372] Lustre: server umount lustre-MDT0001 complete [ 1922.409276] LustreError: 99475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.421807] LustreError: 99475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1923.996962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.884734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1928.949447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1928.954174] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1929.011614] Lustre: 98739:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1930.733977] LustreError: 102603:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1930.742228] LustreError: 102603:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1930.752624] LustreError: 98743:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1930.875461] LustreError: 102612:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -17 [ 1930.896757] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1930.896757] [ 1932.487963] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1934.831350] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1938.240466] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 19:31:01 (1763512261) [ 1950.057935] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1961.916516] Lustre: 104602:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1977.387186] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1979.702927] Lustre: 105735:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2057.696965] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.699196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.713636] Lustre: Skipped 4 previous similar messages [ 2057.716379] Lustre: Skipped 5 previous similar messages [ 2062.306525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2063.788493] LustreError: 106188:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2063.792142] LustreError: 106188:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2063.857784] Lustre: server umount lustre-MDT0000 complete [ 2066.451668] LustreError: 101730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512389 with bad export cookie 584390908930219554 [ 2066.459559] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.467772] LustreError: 101730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2066.782667] Lustre: server umount lustre-MDT0001 complete [ 2080.103158] Lustre: server umount lustre-OST0000 complete [ 2093.808953] Lustre: server umount lustre-OST0001 complete [ 2098.564553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2105.188422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2105.928222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2111.948025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2117.415285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2118.108115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2132.335823] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2140.319490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.334350] Lustre: lustre-MDT0000: reset Object Index mappings [ 2140.686689] LustreError: 109670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.714695] LustreError: 109670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2140.792318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.799762] Lustre: Skipped 1 previous similar message [ 2144.072862] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2149.798129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2149.807696] Lustre: lustre-MDT0001: reset Object Index mappings [ 2153.177493] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2155.173274] Lustre: 110786:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2159.623295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2161.267456] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2164.774119] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2166.399598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2171.419591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2173.381696] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 2173.393494] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2176.004508] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2181.985426] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2184.483674] Lustre: 112640:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2223.462236] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:35:46 (1763512546) [ 2252.772675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2252.778530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.783969] Lustre: Skipped 1 previous similar message [ 2252.786940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.790785] Lustre: Skipped 4 previous similar messages [ 2256.608704] LustreError: 112994:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2256.610675] LustreError: 112994:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2256.654264] Lustre: server umount lustre-MDT0000 complete [ 2258.826975] LustreError: 109651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763512582 with bad export cookie 584390908930266426 [ 2258.828703] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.834184] LustreError: 109651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2258.919921] LustreError: 109665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.931497] LustreError: 109665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2273.253312] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2273.591580] Lustre: server umount lustre-MDT0001 complete [ 2291.169434] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2291.269337] Lustre: server umount lustre-OST0000 complete [ 2300.639789] Lustre: server umount lustre-OST0001 complete [ 2304.734283] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2311.897784] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2318.761356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2323.568955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2328.031177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2333.550505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2339.048413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2339.113371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.265607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2339.292556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2339.346829] Lustre: lustre-MDT0000: new disk, initializing [ 2339.434814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.438391] Lustre: Skipped 3 previous similar messages [ 2339.446682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2341.878687] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2346.508184] LustreError: 116352:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2346.511346] LustreError: 116352:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2346.602728] Lustre: server umount lustre-MDT0000 complete [ 2348.467584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2353.094890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2353.135526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.350040] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2353.361836] Lustre: 116871:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2353.368804] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2355.925263] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2359.089061] Lustre: server umount lustre-MDT0000 complete [ 2364.311466] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:38:07 (1763512687) [ 2365.353176] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2366.405196] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:38:09 (1763512689) [ 2374.822311] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2381.006712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.775364] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2389.006450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2389.042069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2389.114683] Lustre: 118889:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2389.169775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2389.172934] Lustre: Skipped 1 previous similar message [ 2389.251785] Lustre: lustre-MDT0001: new disk, initializing [ 2389.355649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2389.365332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2391.624402] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2394.899820] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2398.364947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2398.404082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2398.551880] Lustre: lustre-OST0000: new disk, initializing [ 2398.555309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2400.525286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2400.532457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2400.596059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2401.659852] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2408.397067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2408.458888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2408.562861] Lustre: lustre-OST0001: new disk, initializing [ 2408.567672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2410.052764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2410.058442] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2410.134061] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2412.263266] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2419.130718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2421.563121] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2732.628940] Lustre: 127684:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2732.638250] Lustre: 127684:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2732.642836] Lustre: 127684:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2732.657534] Lustre: 127684:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2732.670526] Lustre: 127684:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2732.674584] Lustre: 127684:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2977.247196] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:48:20 (1763513300) [ 2992.106631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.109359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.111752] Lustre: Skipped 7 previous similar messages [ 2992.117404] Lustre: Skipped 11 previous similar messages [ 2997.184456] LustreError: 127912:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2997.188140] LustreError: 127912:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2997.219322] LustreError: 122077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.245665] LustreError: 122077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2997.393813] Lustre: server umount lustre-MDT0000 complete [ 3000.233414] LustreError: 118881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763513323 with bad export cookie 584390908930328985 [ 3000.245456] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3002.341889] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.348049] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3002.357307] Lustre: Skipped 2 previous similar messages [ 3002.359787] Lustre: Skipped 2 previous similar messages [ 3005.345320] LustreError: 118896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.354505] LustreError: 118896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3006.639624] Lustre: server umount lustre-MDT0001 complete [ 3015.672347] Lustre: server umount lustre-OST0000 complete [ 3024.706491] Lustre: server umount lustre-OST0001 complete [ 3029.714977] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3038.963590] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3048.499763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.925904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3059.267723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3064.341896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3068.941383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3068.991262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.184799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3069.214838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3069.262993] Lustre: lustre-MDT0000: new disk, initializing [ 3069.351762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.355338] Lustre: Skipped 5 previous similar messages [ 3069.367311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3072.077536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3080.545170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3080.593394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3080.669758] Lustre: 130703:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3080.705135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3080.707970] Lustre: Skipped 1 previous similar message [ 3080.757720] Lustre: lustre-MDT0001: new disk, initializing [ 3080.893488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3080.905518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3084.057866] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3090.663879] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3093.765229] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3099.262226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3099.317231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3099.532677] Lustre: lustre-OST0000: new disk, initializing [ 3099.543207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3101.310030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3101.327661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3101.440156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3104.566731] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3110.594501] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3117.179213] Lustre: lustre-MDD0000: changelog on [ 3119.244184] Lustre: lustre-MDD0001: changelog on [ 4573.508115] Lustre: lustre-MDD0001: changelog off [ 4574.870220] Lustre: 130713:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4575.544445] Lustre: lustre-MDD0000: changelog off [ 4581.357449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.357770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4581.372049] Lustre: Skipped 1 previous similar message [ 4581.372522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4581.372526] Lustre: Skipped 3 previous similar messages [ 4581.376109] LustreError: Skipped 1 previous similar message [ 4585.732113] LustreError: 140478:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4585.735270] LustreError: 140478:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4585.854921] Lustre: server umount lustre-OST0000 complete [ 4591.593089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.594398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4591.615631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4591.619221] Lustre: Skipped 2 previous similar messages [ 4595.129581] Lustre: server umount lustre-MDT0000 complete [ 4596.707439] LustreError: 130713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.714814] LustreError: 130713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4597.828622] LustreError: 140682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763514921 with bad export cookie 584390908931140068 [ 4597.829755] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4597.836932] LustreError: 140682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4598.109462] Lustre: server umount lustre-MDT0001 complete [ 4606.041947] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4608.175179] Key type lgssc unregistered [ 4608.407815] LNet: 141414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4608.420827] LNetError: 141414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4608.434122] LNet: Removed LNI 192.168.203.104@tcp [ 4609.076328] Key type .llcrypt unregistered [ 4609.082258] Key type ._llcrypt unregistered [ 4623.588384] Key type ._llcrypt registered [ 4623.589908] Key type .llcrypt registered [ 4623.670419] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4635.483409] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4636.274280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4636.292265] alg: No test for adler32 (adler32-zlib) [ 4637.326419] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 4637.555945] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4639.216154] Key type lgssc registered [ 4640.248212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4647.435190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.106892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.183314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4663.329074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4668.923554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4668.998836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4669.026625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.297121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4670.328598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4670.397681] Lustre: lustre-MDT0000: new disk, initializing [ 4670.460316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.471381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4673.491572] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4681.247839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4681.295875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4681.350543] Lustre: 144570:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4681.401076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4681.403958] Lustre: Skipped 1 previous similar message [ 4681.463418] Lustre: lustre-MDT0001: new disk, initializing [ 4681.557422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4681.593180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4681.600954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4684.172669] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4689.160608] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4691.448948] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4695.131410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4695.166115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4695.297930] Lustre: lustre-OST0000: new disk, initializing [ 4695.301038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4695.356982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4698.530571] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4702.493803] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4703.770714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4703.779036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4703.836383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4706.639547] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_module obdecho/obdecho [ 4707.719038] LustreError: 147189:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4707.724334] LustreError: 147189:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4708.623789] LustreError: 147285:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4717.537588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4717.552197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4717.569599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4719.076332] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.076632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4721.129922] LustreError: 147627:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4721.200071] Lustre: server umount lustre-MDT0000 complete [ 4724.030784] LustreError: 146325:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515047 with bad export cookie 6200874407782087285 [ 4724.032915] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4724.038993] LustreError: 146325:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4724.151967] LustreError: 147829:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4724.162294] LustreError: 147829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4724.194874] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.207716] Lustre: Skipped 1 previous similar message [ 4724.274364] Lustre: server umount lustre-MDT0001 complete [ 4733.065624] LustreError: 148029:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4733.076917] LustreError: 148029:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4733.268396] Lustre: server umount lustre-OST0000 complete [ 4747.169504] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4753.209203] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4760.443924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4764.760262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4768.563854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4772.675470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.415842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.459670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.610091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4777.631461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4777.679915] Lustre: lustre-MDT0000: new disk, initializing [ 4777.746982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4777.772521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4780.392923] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4787.517609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4787.567523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4787.624120] Lustre: 150975:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4787.668463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4787.671384] Lustre: Skipped 1 previous similar message [ 4787.720554] Lustre: lustre-MDT0001: new disk, initializing [ 4787.800549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4787.834761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4787.842457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4790.236297] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4795.320157] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4797.614794] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4802.178689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4802.226472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4802.412437] Lustre: lustre-OST0000: new disk, initializing [ 4802.421968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4802.531423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4803.935020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4803.945561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4804.028930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4806.810592] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4812.635566] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4816.217479] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4816.404134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4819.882429] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4820.066789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4824.547421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4824.550187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4824.552734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4824.555181] Lustre: Skipped 1 previous similar message [ 4824.561118] Lustre: Skipped 3 previous similar messages [ 4829.672058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4834.790927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4834.798026] Lustre: Skipped 2 previous similar messages [ 4836.320317] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4836.369422] LustreError: 153912:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4836.380516] LustreError: 153912:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4836.463845] Lustre: server umount lustre-OST0000 complete [ 4839.906710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.025356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4845.028270] Lustre: Skipped 2 previous similar messages [ 4845.289910] LustreError: 154115:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4845.293259] LustreError: 154115:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4845.380822] Lustre: server umount lustre-MDT0000 complete [ 4848.213326] LustreError: 150967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515171 with bad export cookie 6200874407782088244 [ 4848.218966] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4848.227729] LustreError: 150967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4848.603536] Lustre: server umount lustre-MDT0001 complete [ 4851.634660] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:19:34 (1763515174) [ 4852.967884] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4854.306280] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:19:37 (1763515177) [ 4859.434032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.804672] LustreError: 154987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.817129] LustreError: 154987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4859.870168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.818813] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4865.002671] LustreError: 154987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4869.595565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4872.945361] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4876.998960] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4879.752710] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4884.492624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4888.544821] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4892.548673] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4903.288068] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 4903.291063] Lustre: Skipped 1 previous similar message [ 4904.074690] LustreError: 157234:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4904.080905] LustreError: 157234:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4904.212592] Lustre: server umount lustre-MDT0000 complete [ 4905.441044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4905.445765] LustreError: 154981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.469024] Lustre: Skipped 3 previous similar messages [ 4905.477446] LustreError: 154981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4906.958750] LustreError: 156605:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515230 with bad export cookie 6200874407782089126 [ 4906.965256] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4906.972876] LustreError: 156605:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4908.396717] LustreError: 154982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.509336] LustreError: 154983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.524752] LustreError: 154983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4921.314808] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4921.400574] LustreError: 157435:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4921.405289] LustreError: 157435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4921.500774] Lustre: server umount lustre-MDT0001 complete [ 4925.918084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4933.951728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4934.121723] LustreError: 158495:0:(llog.c:1616:llog_backup()) MGC192.168.203.104@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4934.136921] Lustre: 158495:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.104@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4947.426591] LustreError: 158501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.435661] LustreError: 158501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4964.709187] LustreError: 158502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.731510] LustreError: 158502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4995.040217] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4995.048047] Lustre: 158495:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4997.602628] LustreError: 158521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4997.620731] LustreError: 158521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 5025.760347] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5025.764869] Lustre: 154978:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5036.016494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5036.025737] Lustre: Skipped 2 previous similar messages [ 5038.419201] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5041.129486] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5043.784194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.240267] LustreError: 154978:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5052.386064] LustreError: 142780:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f9cb166e x1849179462811136/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5054.971062] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5058.025580] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5058.422762] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5059.465487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5061.656812] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5066.723956] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5083.201240] LustreError: 160004:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5083.204725] LustreError: 160004:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5083.272677] Lustre: server umount lustre-MDT0000 complete [ 5083.620588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.622308] LustreError: 156592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.639021] Lustre: Skipped 2 previous similar messages [ 5083.645978] LustreError: 156592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 5085.516089] LustreError: 154967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515408 with bad export cookie 6200874407782095608 [ 5085.520119] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5085.521464] LustreError: 154967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5085.769387] Lustre: server umount lustre-MDT0001 complete [ 5098.405556] Lustre: server umount lustre-OST0000 complete [ 5112.298088] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5117.761205] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5123.289680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5127.424152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5131.436989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5135.616458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5140.718052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5140.756279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.918390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5140.947465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5140.991694] Lustre: lustre-MDT0000: new disk, initializing [ 5141.060100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.062531] Lustre: Skipped 1 previous similar message [ 5141.077373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5143.628541] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5150.792638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5150.830826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5150.882281] Lustre: 163353:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5151.039210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5153.594231] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5158.929800] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5161.803274] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5166.819627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5166.860526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5170.733637] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5172.259024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5172.334674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5175.822582] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5179.004889] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5179.139972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5181.812352] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5181.948470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5187.554609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.555851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5187.556247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5187.556251] Lustre: Skipped 6 previous similar messages [ 5187.561641] Lustre: Skipped 2 previous similar messages [ 5187.566835] LustreError: Skipped 1 previous similar message [ 5189.235379] LustreError: 166292:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5189.238571] LustreError: 166292:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5189.293656] Lustre: server umount lustre-OST0000 complete [ 5192.163082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5197.511611] Lustre: server umount lustre-MDT0000 complete [ 5199.364956] LustreError: 163345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515522 with bad export cookie 6200874407782112422 [ 5199.365827] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5199.378190] LustreError: 163345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5199.539618] Lustre: server umount lustre-MDT0001 complete [ 5204.362034] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:25:27 (1763515527) [ 5208.506218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5211.771452] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5214.179331] LustreError: 167365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5214.188389] LustreError: 167365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5216.880885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5219.482205] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5223.081125] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5225.207651] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5229.304750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5233.032465] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5236.113054] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5251.246242] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:26:14 (1763515574) [ 5255.138333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.142225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.152800] Lustre: Skipped 2 previous similar messages [ 5255.158141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.161197] Lustre: Skipped 3 previous similar messages [ 5259.375817] Lustre: server umount lustre-OST0000 complete [ 5276.128129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5276.250441] Lustre: server umount lustre-MDT0000 complete [ 5278.653854] LustreError: 167345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515601 with bad export cookie 6200874407782113304 [ 5278.659743] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5278.665355] LustreError: 167345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5278.898552] Lustre: server umount lustre-MDT0001 complete [ 5286.417737] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5288.309837] Key type lgssc unregistered [ 5288.524733] LNet: 170760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5288.529925] LNetError: 170760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5288.540440] LNet: Removed LNI 192.168.203.104@tcp [ 5289.039663] Key type .llcrypt unregistered [ 5289.041325] Key type ._llcrypt unregistered [ 5299.942395] Key type ._llcrypt registered [ 5299.944321] Key type .llcrypt registered [ 5300.032978] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5300.484596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5300.509749] alg: No test for adler32 (adler32-zlib) [ 5301.463144] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 5301.630270] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5303.264180] Key type lgssc registered [ 5303.864683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5309.136884] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5309.151691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5324.641716] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5329.760161] LustreError: 171961:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 5334.880451] LustreError: 171961:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5334.898599] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5334.908418] LustreError: MGC192.168.203.104@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5334.914272] LustreError: 171961:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5334.925690] LustreError: 171961:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 5334.931336] LustreError: 171961:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5334.945244] LustreError: 171961:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5334.997253] Lustre: server umount lustre-OST0000 complete [ 5334.999317] LustreError: 171961:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5341.491523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.811757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.455875] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5350.910292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5351.162938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5353.779129] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5357.118116] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5359.362953] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5363.620404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5363.738284] LustreError: 174053:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5363.745339] Lustre: 174053:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.203.104@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5363.845563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.380303] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5370.879300] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5373.417522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.421112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5373.442463] Lustre: Skipped 1 previous similar message [ 5376.545857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5376.548516] Lustre: Skipped 2 previous similar messages [ 5378.517777] LustreError: 174706:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5378.522012] LustreError: 174706:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5378.624550] Lustre: server umount lustre-MDT0000 complete [ 5380.821251] LustreError: 172444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515704 with bad export cookie 11591777985700632046 [ 5380.826601] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5380.831778] LustreError: 172444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5381.602797] LustreError: 172458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.605326] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.620067] LustreError: 172458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5381.625603] Lustre: Skipped 1 previous similar message [ 5381.635122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5381.640496] Lustre: Skipped 1 previous similar message [ 5382.690439] LustreError: 173178:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000055811500 x1849180158974336/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5382.719517] LustreError: 174908:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5382.722729] LustreError: 174908:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5382.800698] Lustre: server umount lustre-MDT0001 complete [ 5385.002040] LustreError: 175112:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5385.004774] LustreError: 175112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5385.075494] Lustre: server umount lustre-OST0000 complete [ 5390.474620] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5396.109869] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5401.755564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5405.250481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5408.875850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.387078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5416.028295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5416.063717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.173752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5416.190287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5416.231122] Lustre: lustre-MDT0000: new disk, initializing [ 5416.279490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.289590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5418.217079] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5424.696308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5424.728655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5424.779709] Lustre: 177443:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5424.818410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5424.821232] Lustre: Skipped 1 previous similar message [ 5424.865759] Lustre: lustre-MDT0001: new disk, initializing [ 5424.931735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5424.960792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5424.966836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5427.134536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5431.893407] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5433.919039] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5437.939775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5437.993846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5438.127977] Lustre: lustre-OST0000: new disk, initializing [ 5438.132437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5438.185438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.320157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5438.324667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5438.439944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5442.156366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5446.904868] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5449.024637] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5449.146641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5451.435844] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5451.576795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5453.796023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.799287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5453.808976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5453.817918] Lustre: Skipped 1 previous similar message [ 5458.749233] LustreError: 180385:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5458.752382] LustreError: 180385:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5458.818137] Lustre: server umount lustre-OST0000 complete [ 5464.034870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.036363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5464.042621] Lustre: Skipped 2 previous similar messages [ 5474.274377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5474.276662] Lustre: Skipped 4 previous similar messages [ 5475.296122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5475.366635] LustreError: 180587:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5475.369768] LustreError: 180587:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5475.413703] Lustre: server umount lustre-MDT0000 complete [ 5477.604430] LustreError: 180588:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515800 with bad export cookie 11591777985700632760 [ 5477.605217] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5477.609698] LustreError: 180588:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5477.837327] Lustre: server umount lustre-MDT0001 complete [ 5482.897595] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:30:06 (1763515806) [ 5491.490421] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5492.974332] Key type lgssc unregistered [ 5493.153828] LNet: 181974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5493.157865] LNetError: 181974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5493.169101] LNet: Removed LNI 192.168.203.104@tcp [ 5493.611562] Key type .llcrypt unregistered [ 5493.613440] Key type ._llcrypt unregistered [ 5504.032471] Key type ._llcrypt registered [ 5504.036295] Key type .llcrypt registered [ 5504.133604] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5504.693428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5504.707143] alg: No test for adler32 (adler32-zlib) [ 5505.629847] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 5505.846585] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5507.464180] Key type lgssc registered [ 5508.201961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5513.140356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5513.154243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5514.444735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5516.728478] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5522.156896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5522.408238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5524.767744] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5528.234107] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5530.469760] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5535.484972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5535.611930] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5535.617035] LustreError: 184789:0:(llog.c:1657:llog_backup()) MGC192.168.203.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5535.622681] Lustre: 184789:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.203.104@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5535.709491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5539.321577] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5542.546906] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5543.907706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5543.909878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5543.916838] Lustre: Skipped 1 previous similar message [ 5543.922558] Lustre: Skipped 2 previous similar messages [ 5546.980924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5546.989205] Lustre: Skipped 2 previous similar messages [ 5552.103171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5552.106860] Lustre: Skipped 2 previous similar messages [ 5557.219544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5557.224507] Lustre: Skipped 2 previous similar messages [ 5558.240122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5558.301125] LustreError: 185440:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5558.313735] LustreError: 185440:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c70d911f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5558.342271] Lustre: server umount lustre-MDT0000 complete [ 5560.338663] LustreError: 183133:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515883 with bad export cookie 1721560745020016162 [ 5560.346131] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5560.346702] LustreError: 183133:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5561.387984] LustreError: 183865:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000adc9d76b x1849180373933824/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5561.406574] LustreError: 185643:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5561.410126] LustreError: 185643:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5561.479839] Lustre: server umount lustre-MDT0001 complete [ 5572.861164] LustreError: 185847:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5572.863737] LustreError: 185847:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5572.926625] Lustre: server umount lustre-OST0000 complete [ 5577.255267] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5582.589766] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5587.325863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5590.636917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5593.620247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5596.648471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5600.058934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5600.104977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.236255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5600.256248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5600.303216] Lustre: lustre-MDT0000: new disk, initializing [ 5600.347906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.362198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5602.299975] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5608.249108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5608.282933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5608.325975] Lustre: 188182:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5608.352113] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5608.353897] Lustre: Skipped 1 previous similar message [ 5608.355593] LustreError: 188948:0:(llog.c:1657:llog_backup()) MGC192.168.203.104@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5608.360980] LustreError: 188948:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5608.363710] Lustre: 188948:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.104@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5608.377360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5608.379921] Lustre: Skipped 1 previous similar message [ 5608.435568] Lustre: lustre-MDT0001: new disk, initializing [ 5608.499245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5608.520520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5608.526967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5610.611616] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5614.939691] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5617.108024] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5621.054519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5621.094848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5621.234634] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5621.237113] Lustre: Skipped 1 previous similar message [ 5621.239723] LustreError: 189901:0:(llog.c:1657:llog_backup()) MGC192.168.203.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5621.244625] LustreError: 189901:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5621.248086] Lustre: 189901:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.104@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5621.251645] Lustre: 189901:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5621.261951] Lustre: lustre-OST0000: new disk, initializing [ 5621.264992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5621.313287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.891902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5623.897787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5623.970351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5624.653925] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5628.995961] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5631.031485] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5631.134842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5632.866733] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5632.962547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5634.532570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5634.532639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5634.538226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5634.542845] Lustre: Skipped 2 previous similar messages [ 5644.770990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5644.774020] Lustre: Skipped 4 previous similar messages [ 5648.352180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5648.381804] LustreError: 191120:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5648.384770] LustreError: 191120:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5648.445288] Lustre: server umount lustre-OST0000 complete [ 5654.500768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5654.504689] LustreError: Skipped 1 previous similar message [ 5654.506801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.269690] LustreError: 191322:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5656.272125] LustreError: 191322:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5656.312904] Lustre: server umount lustre-MDT0000 complete [ 5657.907878] LustreError: 191323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763515981 with bad export cookie 1721560745020016883 [ 5657.911430] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5657.913988] LustreError: 191323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5658.042326] Lustre: server umount lustre-MDT0001 complete [ 5662.033617] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:33:05 (1763515985) [ 5669.729639] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5674.267142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.505022] LustreError: 192891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5674.516250] LustreError: 192891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5674.538397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.500403] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5679.585594] LustreError: 192891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.504423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5682.689061] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5684.181195] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5687.536970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5690.789106] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5694.781287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5694.817410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5694.902551] Lustre: lustre-OST0001: new disk, initializing [ 5694.905647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5697.648200] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5698.145996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5698.150223] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5698.216709] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5704.237885] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5707.728274] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5713.587079] Lustre: DEBUG MARKER: ADD OST3 [ 5715.730202] loop: module loaded [ 5715.752970] loop0: detected capacity change from 0 to 204800000 [ 5715.855884] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5715.863206] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5715.868852] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5715.939635] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5718.467278] loop0: detected capacity change from 0 to 204800000 [ 5720.927590] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5720.957411] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5721.043932] Lustre: lustre-OST0003: new disk, initializing [ 5721.046940] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5721.121069] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5721.122343] Lustre: DEBUG MARKER: STOP OST3 [ 5721.123816] Lustre: Skipped 3 previous similar messages [ 5721.947953] Lustre: Failing over lustre-OST0003 [ 5721.972996] LustreError: 197104:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5721.976516] LustreError: 197104:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5722.012622] Lustre: server umount lustre-OST0003 complete [ 5722.497551] LustreError: 194343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.796570] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5727.594262] LustreError: 194345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.601443] LustreError: 194345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5727.898990] Lustre: DEBUG MARKER: START OST3 again [ 5730.940082] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5732.987079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5732.991769] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5733.005477] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5733.441444] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5748.244527] Lustre: Failing over lustre-OST0003 [ 5748.267168] LustreError: 199180:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 5748.269176] LustreError: 199180:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5748.292635] Lustre: server umount lustre-OST0003 complete [ 5748.706163] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5748.708089] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5748.708817] LustreError: Skipped 1 previous similar message [ 5748.718333] Lustre: Skipped 1 previous similar message [ 5748.720413] LustreError: 194345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5748.727477] LustreError: 194345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5756.900306] LustreError: 197241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5756.906968] LustreError: 197241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5762.021697] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.024754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5762.027870] Lustre: Skipped 3 previous similar messages [ 5762.030585] Lustre: Skipped 5 previous similar messages [ 5767.808565] Lustre: server umount lustre-MDT0000 complete [ 5769.502354] LustreError: 192871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516092 with bad export cookie 1721560745020017765 [ 5769.506812] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.507596] LustreError: 192871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5772.264211] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.269259] Lustre: Skipped 1 previous similar message [ 5774.304996] LustreError: 193955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.313164] LustreError: 193955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5775.789179] Lustre: server umount lustre-MDT0001 complete [ 5777.733382] Lustre: server umount lustre-OST0000 complete [ 5779.575922] Lustre: server umount lustre-OST0001 complete [ 5782.370330] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5786.086766] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5790.876922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5794.216210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5797.502044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5800.766872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5803.906230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5803.940436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.059085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5804.079576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5804.124686] Lustre: lustre-MDT0000: new disk, initializing [ 5804.166639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.169822] Lustre: Skipped 1 previous similar message [ 5804.178319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5805.911587] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5811.586621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5811.637413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5811.698502] Lustre: 202234:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5811.855215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5813.686819] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5817.771982] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5819.442441] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5822.581752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5822.615962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5822.742848] Lustre: lustre-OST0000: new disk, initializing [ 5822.745656] Lustre: Skipped 1 previous similar message [ 5822.751831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5822.756510] Lustre: Skipped 2 previous similar messages [ 5824.339995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5824.344554] Lustre: Skipped 1 previous similar message [ 5824.347315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5824.402217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5825.443539] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5829.861215] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5831.705277] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5831.808824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5833.718357] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5833.816726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5834.723410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5834.723552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.726612] LustreError: Skipped 1 previous similar message [ 5834.733349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5834.737391] Lustre: Skipped 9 previous similar messages [ 5840.756107] LustreError: 205173:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5840.758075] LustreError: 205173:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 5840.804454] Lustre: server umount lustre-OST0000 complete [ 5848.665739] Lustre: server umount lustre-MDT0000 complete [ 5850.081101] LustreError: 202246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5850.091158] LustreError: 202246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5850.476606] LustreError: 202227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516173 with bad export cookie 1721560745020020026 [ 5850.479198] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5850.483357] LustreError: 202227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5850.653521] Lustre: server umount lustre-MDT0001 complete [ 5852.433990] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:36:16 (1763516176) [ 5859.540302] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5863.640758] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5867.849975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5870.853438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5873.575651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5876.744929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5880.197624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5880.227691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.365092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5880.384300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5880.425300] Lustre: lustre-MDT0000: new disk, initializing [ 5880.480069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5882.273873] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5887.689669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5887.724886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5887.773218] Lustre: 208416:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5887.921367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5889.835567] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5893.811471] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5895.492539] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5898.511868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5898.544274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5900.233257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5901.125455] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5904.879678] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5906.686869] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5906.791891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5908.525540] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5908.625068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5910.497963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5910.500664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.503255] LustreError: Skipped 1 previous similar message [ 5910.510697] Lustre: Skipped 3 previous similar messages [ 5910.513966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5910.518254] Lustre: Skipped 5 previous similar messages [ 5915.506129] LustreError: 211352:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5915.523867] LustreError: 211352:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5915.571546] Lustre: server umount lustre-OST0000 complete [ 5918.690860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5923.252782] Lustre: server umount lustre-MDT0000 complete [ 5924.965526] LustreError: 208406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516248 with bad export cookie 1721560745020020901 [ 5924.968726] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.971533] LustreError: 208406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5925.121407] Lustre: server umount lustre-MDT0001 complete [ 5929.313980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.566705] LustreError: 212232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.359921] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5934.984856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5935.172218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5935.175509] Lustre: Skipped 6 previous similar messages [ 5936.817662] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5939.015907] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5940.518077] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5943.310828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5945.762949] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5948.042574] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5961.185074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.382311] Lustre: server umount lustre-OST0000 complete [ 5974.262518] Lustre: server umount lustre-MDT0000 complete [ 5975.914559] LustreError: 212213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516299 with bad export cookie 1721560745020021790 [ 5975.918875] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5975.920031] LustreError: 212213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5976.083784] Lustre: server umount lustre-MDT0001 complete [ 6080.967427] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6082.073199] Key type lgssc unregistered [ 6082.208747] LNet: 215562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6082.211991] LNetError: 215562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6082.220354] LNet: Removed LNI 192.168.203.104@tcp [ 6082.536217] Key type .llcrypt unregistered [ 6082.537815] Key type ._llcrypt unregistered [ 6092.449561] Key type ._llcrypt registered [ 6092.450740] Key type .llcrypt registered [ 6092.498056] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6099.389715] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6099.865508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6099.876256] alg: No test for adler32 (adler32-zlib) [ 6100.742028] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6100.836883] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6102.424162] Key type lgssc registered [ 6102.924069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6106.552871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6109.378111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6111.879123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6114.353250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6117.050167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6117.071189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6117.080610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.177638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6118.197724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6118.244202] Lustre: lustre-MDT0000: new disk, initializing [ 6118.288946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6118.298584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6119.840559] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6124.682492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6124.711275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6124.755440] Lustre: 218863:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6124.782421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6124.785224] Lustre: Skipped 1 previous similar message [ 6124.824198] Lustre: lustre-MDT0001: new disk, initializing [ 6124.865737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6124.884619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6124.888719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6126.398827] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6129.817681] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6131.196054] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6133.619089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6133.648044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6133.745301] Lustre: lustre-OST0000: new disk, initializing [ 6133.748351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6133.779563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6135.035867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6135.039389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6135.072711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6135.773804] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6139.131942] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6140.661519] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6140.742836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6142.208343] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6142.295937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6145.505839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6145.506545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.510939] LustreError: Skipped 1 previous similar message [ 6145.515927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6145.521321] Lustre: Skipped 1 previous similar message [ 6150.627060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6150.629957] Lustre: Skipped 1 previous similar message [ 6155.746577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6155.749194] Lustre: Skipped 1 previous similar message [ 6157.280154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6157.300094] LustreError: 221802:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6157.343597] Lustre: server umount lustre-OST0000 complete [ 6160.866267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6160.866929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.867839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6160.876387] Lustre: Skipped 2 previous similar messages [ 6165.986656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6165.989330] Lustre: Skipped 2 previous similar messages [ 6173.152285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6173.192337] LustreError: 222005:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6173.194266] LustreError: 222005:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6173.225801] Lustre: server umount lustre-MDT0000 complete [ 6174.729905] LustreError: 219642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516498 with bad export cookie 9208659354887687209 [ 6174.733131] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6174.734060] LustreError: 219642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6174.812702] LustreError: 222207:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6174.816561] LustreError: 222207:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6174.882147] Lustre: server umount lustre-MDT0001 complete [ 6181.915202] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6185.849671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.025956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6187.509331] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6190.688799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6190.851579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6192.387414] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6193.531735] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6196.123069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6198.603525] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6202.099179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6202.130035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6202.196929] Lustre: lustre-OST0001: new disk, initializing [ 6202.199517] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6202.236601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6202.239495] Lustre: Skipped 1 previous similar message [ 6204.396899] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6207.500857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6207.505494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6207.538476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6214.769061] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6222.017251] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6231.617501] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:42:35 (1763516555) [ 6237.153349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6237.155521] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.156390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6237.159297] Lustre: Skipped 2 previous similar messages [ 6237.162534] Lustre: Skipped 5 previous similar messages [ 6238.400351] LustreError: 226758:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6238.403585] LustreError: 226758:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6238.430032] Lustre: server umount lustre-MDT0000 complete [ 6239.694459] LustreError: 223360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516563 with bad export cookie 9208659354887688077 [ 6239.697447] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6239.698079] LustreError: 223360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6239.817194] Lustre: server umount lustre-MDT0001 complete [ 6251.448096] LustreError: 227160:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6251.450049] LustreError: 227160:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6251.467924] Lustre: server umount lustre-OST0000 complete [ 6262.968137] LustreError: 227362:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6262.970632] LustreError: 227362:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6263.021734] Lustre: server umount lustre-OST0001 complete [ 6265.543871] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6269.082377] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6272.713283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6275.239671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6277.807889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6280.450984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6283.250108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6283.280123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.380184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6283.395384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6283.435557] Lustre: lustre-MDT0000: new disk, initializing [ 6283.469191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6283.478608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6285.134496] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6290.019153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6290.045962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6290.087678] Lustre: 229545:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6290.208554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6291.768139] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6295.322312] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6296.754802] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6299.269172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6299.294318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6299.406513] Lustre: lustre-OST0000: new disk, initializing [ 6299.407908] Lustre: Skipped 1 previous similar message [ 6299.410685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6299.413484] Lustre: Skipped 2 previous similar messages [ 6300.991680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6300.995609] Lustre: Skipped 1 previous similar message [ 6300.997926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6301.038646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6301.578071] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6305.167273] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6306.762690] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6306.843825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6308.331393] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6308.432029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6311.394425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6311.395114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.399811] LustreError: Skipped 1 previous similar message [ 6311.400604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6311.405314] Lustre: Skipped 2 previous similar messages [ 6315.257100] LustreError: 232484:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6315.259991] LustreError: 232484:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6315.311641] Lustre: server umount lustre-OST0000 complete [ 6321.121112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6321.126369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6331.360209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6331.429430] Lustre: server umount lustre-MDT0000 complete [ 6332.951668] LustreError: 229538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516656 with bad export cookie 9208659354887690205 [ 6332.954719] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6332.957129] LustreError: 229538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6333.111883] Lustre: server umount lustre-MDT0001 complete [ 6336.322888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.503996] LustreError: 233365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.511327] LustreError: 233365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6336.532515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.535571] Lustre: Skipped 2 previous similar messages [ 6338.034099] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6341.529709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6341.601661] LustreError: 233365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.266309] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6345.580088] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6347.237877] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6350.457519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6353.111519] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6355.361485] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6500.321588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6500.324797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.330342] Lustre: Skipped 1 previous similar message [ 6500.333612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6500.336523] Lustre: Skipped 7 previous similar messages [ 6505.711139] LustreError: 235651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6505.714299] LustreError: 235651:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6505.759577] Lustre: server umount lustre-OST0000 complete [ 6513.365041] Lustre: server umount lustre-MDT0000 complete [ 6514.656905] LustreError: 233383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.662795] LustreError: 233383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6514.726402] LustreError: 234084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763516838 with bad export cookie 9208659354887691087 [ 6514.729498] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6514.731457] LustreError: 234084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6514.858472] Lustre: server umount lustre-MDT0001 complete [ 6625.519763] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6626.640279] Key type lgssc unregistered [ 6626.789386] LNet: 236623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6626.792859] LNetError: 236623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6626.800909] LNet: Removed LNI 192.168.203.104@tcp [ 6627.137160] Key type .llcrypt unregistered [ 6627.138433] Key type ._llcrypt unregistered [ 6636.458878] Key type ._llcrypt registered [ 6636.460282] Key type .llcrypt registered [ 6636.508241] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6643.604300] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6644.034692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6644.045312] alg: No test for adler32 (adler32-zlib) [ 6644.904290] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 6644.992319] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6646.576117] Key type lgssc registered [ 6646.951898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6650.078344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.268479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.375476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6656.568042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6662.663692] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6666.630554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6666.655187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6666.661330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.767535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6667.783274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6667.822940] Lustre: lustre-MDT0000: new disk, initializing [ 6667.861605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6667.870074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6669.390497] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6674.061961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6674.092923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6674.136908] Lustre: 240636:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6674.167515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6674.170241] Lustre: Skipped 1 previous similar message [ 6674.210402] Lustre: lustre-MDT0001: new disk, initializing [ 6674.248869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6674.266038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6674.270138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6675.809219] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6678.306965] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6680.752219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6680.780486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6680.890327] Lustre: lustre-OST0000: new disk, initializing [ 6680.893346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6680.933740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6683.028831] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6686.203434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6686.208228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6686.252460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6687.396448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6687.425021] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6687.496387] Lustre: lustre-OST0001: new disk, initializing [ 6687.499300] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6687.543791] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6689.647248] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6693.888295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6693.891493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6693.920552] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6695.249712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6702.193439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6714.339371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.343657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6714.344738] Lustre: Skipped 3 previous similar messages [ 6714.346441] Lustre: Skipped 2 previous similar messages [ 6716.431096] LustreError: 244155:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6716.467663] Lustre: server umount lustre-MDT0000 complete [ 6717.764918] LustreError: 241414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517041 with bad export cookie 10087963796529153074 [ 6717.766226] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.770462] LustreError: 241414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6717.849958] LustreError: 244356:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6717.852380] LustreError: 244356:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6717.921516] Lustre: server umount lustre-MDT0001 complete [ 6729.652072] LustreError: 244556:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6729.654035] LustreError: 244556:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6729.674269] Lustre: server umount lustre-OST0000 complete [ 6741.107826] LustreError: 244758:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6741.109495] LustreError: 244758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6741.153462] Lustre: server umount lustre-OST0001 complete [ 6745.887748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6748.819970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6749.161094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6751.937637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6754.786342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6755.126774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6761.541535] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6764.785296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6764.791486] Lustre: lustre-MDT0000: reset Object Index mappings [ 6764.957969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.270684] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6769.116297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6769.122646] Lustre: lustre-MDT0001: reset Object Index mappings [ 6770.685661] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6771.670843] Lustre: 249176:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6773.815160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6773.936559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.938519] Lustre: Skipped 1 previous similar message [ 6774.949394] LustreError: 249535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6774.952460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6774.955478] LustreError: 249535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6775.741613] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6778.623231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6779.754699] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6780.696314] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6784.004569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6785.354514] Lustre: 251023:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6794.721271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6794.723763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.728641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.565190] LustreError: 251126:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6797.567158] LustreError: 251126:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6797.593069] Lustre: server umount lustre-MDT0000 complete [ 6798.881226] LustreError: 248794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517122 with bad export cookie 10087963796529159850 [ 6798.884569] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6798.885892] LustreError: 248794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6799.005357] Lustre: server umount lustre-MDT0001 complete [ 6810.547112] LustreError: 251528:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6810.549428] LustreError: 251528:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6810.566373] Lustre: server umount lustre-OST0000 complete [ 6822.175362] Lustre: server umount lustre-OST0001 complete [ 6828.768099] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:52:32 (1763517152) [ 6834.063916] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6837.299346] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6840.492609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6842.534301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6844.638966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6846.811126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6849.235808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6849.264046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.356994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6849.373108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6849.410904] Lustre: lustre-MDT0000: new disk, initializing [ 6849.439523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.442293] Lustre: Skipped 1 previous similar message [ 6849.449871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6850.932506] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6855.432684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6855.456872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6855.492343] Lustre: 255247:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6855.601282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6857.013142] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6860.036544] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6861.305324] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6863.701170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6863.726391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6863.830127] Lustre: lustre-OST0000: new disk, initializing [ 6863.832131] Lustre: Skipped 1 previous similar message [ 6863.834919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6863.838233] Lustre: Skipped 2 previous similar messages [ 6865.403803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6865.406826] Lustre: Skipped 1 previous similar message [ 6865.409437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6865.442686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6865.875833] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6869.180094] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6870.640098] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6870.728596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6872.107425] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6872.184540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6875.619485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.620225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6875.624633] Lustre: Skipped 3 previous similar messages [ 6875.625546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.632766] Lustre: Skipped 5 previous similar messages [ 6878.892559] LustreError: 258185:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6878.895847] LustreError: 258185:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6878.938394] Lustre: server umount lustre-OST0000 complete [ 6880.738612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.742895] Lustre: Skipped 1 previous similar message [ 6880.745533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.748382] Lustre: Skipped 1 previous similar message [ 6885.857153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6885.859784] Lustre: Skipped 1 previous similar message [ 6886.487383] Lustre: server umount lustre-MDT0000 complete [ 6887.835513] LustreError: 256029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517211 with bad export cookie 10087963796529161943 [ 6887.839613] LustreError: 256029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6887.840635] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6887.964494] Lustre: server umount lustre-MDT0001 complete [ 6894.580922] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6898.247299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.384703] LustreError: 259760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.399636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6898.402722] Lustre: Skipped 2 previous similar messages [ 6899.697961] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6902.472851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6903.901129] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6904.913191] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6907.207242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6909.304407] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6912.139766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6912.165414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6912.223535] Lustre: lustre-OST0001: new disk, initializing [ 6912.226240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6914.140289] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6914.246889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6914.250858] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6914.283126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6918.647339] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6919.979451] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6926.909452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6926.915921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6929.890753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6929.893397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6929.895967] Lustre: Skipped 4 previous similar messages [ 6929.899045] Lustre: Skipped 2 previous similar messages [ 6934.912405] LustreError: 263264:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6934.915620] LustreError: 263264:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6934.950169] Lustre: server umount lustre-MDT0000 complete [ 6935.009931] LustreError: 260471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6935.015912] LustreError: 260471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6936.225381] LustreError: 260480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517259 with bad export cookie 10087963796529162825 [ 6936.228983] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6936.230063] LustreError: 260480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6936.347189] Lustre: server umount lustre-MDT0001 complete [ 6948.358992] Lustre: server umount lustre-OST0000 complete [ 6958.879320] Lustre: server umount lustre-OST0001 complete [ 6964.200920] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6967.468533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.591128] LustreError: 265043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.610475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.612315] Lustre: Skipped 3 previous similar messages [ 6968.894523] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6971.959797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6973.560085] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6974.542743] Lustre: 266152:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6976.702218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6977.826585] LustreError: 266509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6977.833500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6978.582018] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6981.445151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6982.574852] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6983.491040] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6986.683929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6987.926726] Lustre: 268005:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6994.698253] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6994.708457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6997.722799] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 20:55:21 (1763517321) [ 7013.347674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7013.348709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.349949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.350114] LustreError: Skipped 1 previous similar message [ 7013.353171] Lustre: Skipped 1 previous similar message [ 7015.227939] LustreError: 268712:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7015.229678] LustreError: 268712:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7015.257857] Lustre: server umount lustre-MDT0000 complete [ 7016.414189] LustreError: 265023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517339 with bad export cookie 10087963796529169111 [ 7016.417257] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7016.417888] LustreError: 265023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7018.465140] LustreError: 265071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.471168] LustreError: 265071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7022.570703] Lustre: server umount lustre-MDT0001 complete [ 7029.822285] Lustre: server umount lustre-OST0000 complete [ 7037.270489] Lustre: server umount lustre-OST0001 complete [ 7039.224193] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7042.384770] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7045.672815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7047.702754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7049.660363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7051.768784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7057.228593] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7060.451166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7060.471786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.547273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7060.558614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7060.593123] Lustre: lustre-MDT0000: new disk, initializing [ 7060.622790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7061.897577] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7065.869208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7065.888773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7065.923275] Lustre: 272195:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7066.025158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7067.334544] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7071.862602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7071.890057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7072.101542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 7073.994775] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7078.161088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7078.179540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7080.120675] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7080.195408] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7084.620918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7095.000624] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 20:56:58 (1763517418) [ 7095.780636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.781392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7095.786597] Lustre: Skipped 5 previous similar messages [ 7095.788569] Lustre: Skipped 9 previous similar messages [ 7101.857461] Lustre: server umount lustre-MDT0000 complete [ 7103.163059] LustreError: 272974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517426 with bad export cookie 10087963796529176979 [ 7103.165727] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7103.168224] LustreError: 272974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7103.298325] Lustre: server umount lustre-MDT0001 complete [ 7115.275314] Lustre: server umount lustre-OST0000 complete [ 7125.848612] Lustre: server umount lustre-OST0001 complete [ 7131.601312] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7134.859259] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7138.236984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7140.335230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7142.333819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7144.404021] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7146.929043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7146.948801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7147.038055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7147.041862] Lustre: Skipped 2 previous similar messages [ 7147.051326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7147.053163] Lustre: Skipped 4 previous similar messages [ 7147.088140] Lustre: lustre-MDT0000: new disk, initializing [ 7147.089568] Lustre: Skipped 3 previous similar messages [ 7147.112636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.114546] Lustre: Skipped 7 previous similar messages [ 7147.120850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7147.124014] Lustre: Skipped 3 previous similar messages [ 7148.443106] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7152.495727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7152.516750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7152.548065] Lustre: 279250:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7152.643377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7152.645200] Lustre: Skipped 2 previous similar messages [ 7153.926480] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7157.087840] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7158.302651] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7160.543985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7160.570292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7161.951684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7162.638460] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7165.727762] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7166.427387] Lustre: Mounted lustre-client [ 7166.736259] LustreError: 281752:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9b0c7647d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 7166.741593] LustreError: 281752:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7166.744542] LustreError: 281752:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 7166.764955] Lustre: Unmounted lustre-client [ 7171.555223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.556035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7171.560039] Lustre: Skipped 1 previous similar message [ 7171.562866] Lustre: Skipped 5 previous similar messages [ 7171.731461] Lustre: server umount lustre-OST0000 complete [ 7173.049465] Lustre: server umount lustre-MDT0000 complete [ 7173.089795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7173.093444] LustreError: 279260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.099508] LustreError: 279260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7174.264411] LustreError: 279240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517497 with bad export cookie 10087963796529178505 [ 7174.267810] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7174.268256] LustreError: 279240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7180.468654] Lustre: server umount lustre-MDT0001 complete [ 7184.169072] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7185.181294] Key type lgssc unregistered [ 7222.560106] LNet: 1 local NIs in recovery (showing 1): 192.168.203.104@tcp [ 7223.200305] LNet: 283030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7223.202517] LNetError: 283030:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7296.934332] LNet: Removed LNI 192.168.203.104@tcp [ 7297.232100] Key type .llcrypt unregistered [ 7297.232947] Key type ._llcrypt unregistered [ 7302.965180] Key type ._llcrypt registered [ 7302.966019] Key type .llcrypt registered [ 7303.001565] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7308.876334] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7309.145316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7309.150170] alg: No test for adler32 (adler32-zlib) [ 7309.995746] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7310.079241] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7311.656143] Key type lgssc registered [ 7312.025341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7315.016806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7317.290597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7319.323521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7321.551500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7322.160763] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:00:46 (1763517646) [ 7324.344054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7324.362632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7324.367034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.454879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7325.466690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7325.507724] Lustre: lustre-MDT0000: new disk, initializing [ 7325.532822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7325.539167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7326.984138] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7331.383023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7331.401327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7331.438961] Lustre: 286198:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7331.475404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7331.475636] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7331.477574] Lustre: Skipped 1 previous similar message [ 7331.513761] Lustre: lustre-MDT0001: new disk, initializing [ 7331.549241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7331.564350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7331.568184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7332.973279] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7336.145774] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7337.469557] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7339.889028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7339.914614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7340.008440] Lustre: lustre-OST0000: new disk, initializing [ 7340.010510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7340.038856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7342.055247] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7345.283600] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7346.681686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7346.685292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7346.717424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7356.897701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7356.899711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7356.900460] LustreError: Skipped 1 previous similar message [ 7356.901128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7356.904334] Lustre: Skipped 1 previous similar message [ 7360.496142] LustreError: 288650:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7360.538945] Lustre: server umount lustre-OST0000 complete [ 7362.017589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.018524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.021800] Lustre: Skipped 1 previous similar message [ 7362.025708] Lustre: Skipped 2 previous similar messages [ 7367.137581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7367.139557] Lustre: Skipped 1 previous similar message [ 7372.257016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7372.259481] Lustre: Skipped 1 previous similar message [ 7376.352102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7376.383348] LustreError: 288852:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7376.385174] LustreError: 288852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7376.414801] Lustre: server umount lustre-MDT0000 complete [ 7377.541968] LustreError: 286188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763517700 with bad export cookie 1406281490464716437 [ 7377.544756] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7377.545100] LustreError: 286188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7377.609982] LustreError: 289055:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7377.613022] LustreError: 289055:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7377.663657] Lustre: server umount lustre-MDT0001 complete [ 7381.108796] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7382.066951] Key type lgssc unregistered [ 7382.190359] LNet: 289585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7382.192687] LNetError: 289585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7382.202398] LNet: Removed LNI 192.168.203.104@tcp [ 7382.490818] Key type .llcrypt unregistered [ 7382.492235] Key type ._llcrypt unregistered [ 7414.711618] Key type ._llcrypt registered [ 7414.713048] Key type .llcrypt registered [ 7414.746603] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7414.840326] Key type .llcrypt unregistered [ 7414.841819] Key type ._llcrypt unregistered [ 7439.892374] Key type ._llcrypt registered [ 7439.893382] Key type .llcrypt registered [ 7439.929114] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7440.036366] Key type .llcrypt unregistered [ 7440.037276] Key type ._llcrypt unregistered [ 7450.309265] Key type ._llcrypt registered [ 7450.310504] Key type .llcrypt registered [ 7450.348328] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7450.447381] Key type .llcrypt unregistered [ 7450.448936] Key type ._llcrypt unregistered [ 7460.544262] Key type ._llcrypt registered [ 7460.545529] Key type .llcrypt registered [ 7460.579486] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7460.675305] Key type .llcrypt unregistered [ 7460.676365] Key type ._llcrypt unregistered [ 7486.042379] Key type ._llcrypt registered [ 7486.043925] Key type .llcrypt registered [ 7486.083111] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7486.180396] Key type .llcrypt unregistered [ 7486.181709] Key type ._llcrypt unregistered [ 7496.957459] Key type ._llcrypt registered [ 7496.958828] Key type .llcrypt registered [ 7496.998542] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7497.095325] Key type .llcrypt unregistered [ 7497.096330] Key type ._llcrypt unregistered [ 7507.965592] Key type ._llcrypt registered [ 7507.966614] Key type .llcrypt registered [ 7508.003965] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7508.107419] Key type .llcrypt unregistered [ 7508.108595] Key type ._llcrypt unregistered [ 7519.129878] Key type ._llcrypt registered [ 7519.131304] Key type .llcrypt registered [ 7519.170534] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7519.274333] Key type .llcrypt unregistered [ 7519.276264] Key type ._llcrypt unregistered [ 7530.985686] Key type ._llcrypt registered [ 7530.986845] Key type .llcrypt registered [ 7531.026155] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7531.126201] Key type .llcrypt unregistered [ 7531.127558] Key type ._llcrypt unregistered [ 7541.523847] Key type ._llcrypt registered [ 7541.524764] Key type .llcrypt registered [ 7541.562262] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7541.667523] Key type .llcrypt unregistered [ 7541.669038] Key type ._llcrypt unregistered [ 7562.758937] Key type ._llcrypt registered [ 7562.760217] Key type .llcrypt registered [ 7562.806177] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7562.916396] Key type .llcrypt unregistered [ 7562.917620] Key type ._llcrypt unregistered [ 7573.897045] Key type ._llcrypt registered [ 7573.897874] Key type .llcrypt registered [ 7573.932818] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7574.028235] Key type .llcrypt unregistered [ 7574.029520] Key type ._llcrypt unregistered [ 7584.458536] Key type ._llcrypt registered [ 7584.459889] Key type .llcrypt registered [ 7584.495591] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7584.597288] Key type .llcrypt unregistered [ 7584.598696] Key type ._llcrypt unregistered [ 7602.589551] Key type ._llcrypt registered [ 7602.590517] Key type .llcrypt registered [ 7602.633019] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7608.737780] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7609.024307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7609.053054] alg: No test for adler32 (adler32-zlib) [ 7609.918376] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 7609.999188] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7611.584176] Key type lgssc registered [ 7611.940321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7614.797318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7617.019521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7618.913412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7620.957239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7621.483461] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7399 sec ========== 21:05:45 (1763517945) [ 7622.002933] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:05:45 (1763517945) === [ 7623.118429] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:05:47 (1763517947) === [ 7635.814880] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7636.800574] Key type lgssc unregistered [ 7636.928350] LNet: 311283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7636.931060] LNetError: 311283:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7636.942286] LNet: Removed LNI 192.168.203.104@tcp [ 7637.203575] Key type .llcrypt unregistered [ 7637.204618] Key type ._llcrypt unregistered