[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474678023 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2839944K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003149] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.023069] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027658] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] *** VALIDATE tmpfs *** [ 0.031487] *** VALIDATE proc *** [ 0.033210] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036182] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043099] debug: unmapping init [mem 0xffffffffa7259000-0xffffffffa7260fff] [ 0.045996] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046728] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048017] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054322] rcu: Hierarchical SRCU implementation. [ 0.056512] smp: Bringing up secondary CPUs ... [ 0.057575] x86: Booting SMP configuration: [ 0.058031] .... node #0, CPUs: #1 #2 #3 [ 0.061155] smp: Brought up 1 node, 4 CPUs [ 0.063030] smpboot: Max logical packages: 1 [ 0.064027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139290] node 0 deferred pages initialised in 72ms [ 0.141094] devtmpfs: initialized [ 0.142199] x86/mm: Memory block size: 128MB [ 0.144250] gcov: version magic: 0x41383552 [ 0.146131] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148232] pinctrl core: initialized pinctrl subsystem [ 0.149191] [ 0.149804] ************************************************************* [ 0.150012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151015] ** ** [ 0.152011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153015] ** ** [ 0.154013] ** This means that this kernel is built to expose internal ** [ 0.155014] ** IOMMU data structures, which may compromise security on ** [ 0.156012] ** your system. ** [ 0.157015] ** ** [ 0.158012] ** If you see this message and you are not debugging the ** [ 0.159011] ** kernel, report this immediately to your vendor! ** [ 0.160012] ** ** [ 0.161025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162010] ************************************************************* [ 0.163576] NET: Registered protocol family 16 [ 0.164313] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.167341] cpuidle: using governor menu [ 0.168435] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170395] PCI: Using configuration type 1 for base access [ 0.173104] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181040] cryptd: max_cpu_qlen set to 1000 [ 0.183158] ACPI: Added _OSI(Module Device) [ 0.184016] ACPI: Added _OSI(Processor Device) [ 0.185012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190240] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.196549] ACPI: Interpreter enabled [ 0.198071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200013] ACPI: Using IOAPIC for interrupt routing [ 0.201090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.215931] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229144] acpiphp: Slot [2] registered [ 0.231117] acpiphp: Slot [5] registered [ 0.233104] acpiphp: Slot [6] registered [ 0.234117] acpiphp: Slot [7] registered [ 0.236161] acpiphp: Slot [8] registered [ 0.237106] acpiphp: Slot [9] registered [ 0.239091] acpiphp: Slot [10] registered [ 0.242156] acpiphp: Slot [3] registered [ 0.244101] acpiphp: Slot [4] registered [ 0.246104] acpiphp: Slot [11] registered [ 0.247116] acpiphp: Slot [12] registered [ 0.249113] acpiphp: Slot [13] registered [ 0.250125] acpiphp: Slot [14] registered [ 0.251207] acpiphp: Slot [15] registered [ 0.253107] acpiphp: Slot [16] registered [ 0.255126] acpiphp: Slot [17] registered [ 0.257122] acpiphp: Slot [18] registered [ 0.258124] acpiphp: Slot [19] registered [ 0.260104] acpiphp: Slot [20] registered [ 0.261113] acpiphp: Slot [21] registered [ 0.263120] acpiphp: Slot [22] registered [ 0.265122] acpiphp: Slot [23] registered [ 0.267084] acpiphp: Slot [24] registered [ 0.268095] acpiphp: Slot [25] registered [ 0.270106] acpiphp: Slot [26] registered [ 0.273117] acpiphp: Slot [27] registered [ 0.274095] acpiphp: Slot [28] registered [ 0.276121] acpiphp: Slot [29] registered [ 0.278127] acpiphp: Slot [30] registered [ 0.279088] acpiphp: Slot [31] registered [ 0.281052] PCI host bridge to bus 0000:00 [ 0.282024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302171] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305324] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.316024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.321040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.323021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.325021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.328017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.331376] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.334603] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.337061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.339628] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.355016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.360905] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.366502] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.377026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.432016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.441016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.461015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.471915] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.477017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.484020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.498016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.508000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.522015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.537087] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.552654] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.559017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.565015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.590016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.600715] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.612014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.625017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.639601] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.642236] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.644354] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.647376] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.650293] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.655043] iommu: Default domain type: Passthrough [ 0.656633] SCSI subsystem initialized [ 0.658167] ACPI: bus type USB registered [ 0.660112] usbcore: registered new interface driver usbfs [ 0.662133] usbcore: registered new interface driver hub [ 0.664185] usbcore: registered new device driver usb [ 0.666162] pps_core: LinuxPPS API ver. 1 registered [ 0.669013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.673062] PTP clock support registered [ 0.676058] EDAC MC: Ver: 3.0.0 [ 0.678174] PCI: Using ACPI for IRQ routing [ 0.679860] NetLabel: Initializing [ 0.680022] NetLabel: domain hash size = 128 [ 0.683012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.686084] NetLabel: unlabeled traffic allowed by default [ 0.689078] vgaarb: loaded [ 0.691496] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.702275] clocksource: Switched to clocksource kvm-clock [ 0.807538] VFS: Disk quotas dquot_6.6.0 [ 0.809263] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.812503] *** VALIDATE ramfs *** [ 0.813688] *** VALIDATE hugetlbfs *** [ 0.814713] pnp: PnP ACPI init [ 0.816983] pnp: PnP ACPI: found 6 devices [ 0.832148] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.835555] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.837725] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.840043] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.842482] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.844868] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.847766] NET: Registered protocol family 2 [ 0.850230] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.855421] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.859234] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.864691] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.868770] TCP: Hash tables configured (established 65536 bind 65536) [ 0.871734] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.874993] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.878051] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880330] NET: Registered protocol family 1 [ 0.882668] RPC: Registered named UNIX socket transport module. [ 0.884512] RPC: Registered udp transport module. [ 0.885970] RPC: Registered tcp transport module. [ 0.887668] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.889984] NET: Registered protocol family 44 [ 0.891864] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.894433] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.896755] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.899482] PCI: CLS 0 bytes, default 64 [ 0.901333] Unpacking initramfs... [ 2.300312] debug: unmapping init [mem 0xffff8af2bcc54000-0xffff8af2bffbffff] [ 2.305212] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.307604] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.310459] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.765147] Initialise system trusted keyrings [ 2.766585] Key type blacklist registered [ 2.768628] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.777506] zbud: loaded [ 2.781131] *** VALIDATE nfs *** [ 2.782315] *** VALIDATE nfs4 *** [ 2.783899] pstore: using deflate compression [ 2.789251] Platform Keyring initialized [ 2.883333] NET: Registered protocol family 38 [ 2.884861] Key type asymmetric registered [ 2.886464] Asymmetric key parser 'x509' registered [ 2.888294] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.891058] io scheduler mq-deadline registered [ 2.892528] io scheduler kyber registered [ 2.893892] io scheduler bfq registered [ 2.895174] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.897329] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.899379] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.901537] ACPI: Power Button [PWRF] [ 2.906563] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.911783] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.920391] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.925325] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.936643] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.962576] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.987451] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.991553] Non-volatile memory driver v1.3 [ 2.992633] Linux agpgart interface v0.103 [ 3.019063] virtio_blk virtio1: [vda] 134216 512-byte logical blocks (68.7 MB/65.5 MiB) [ 3.021512] vda: detected capacity change from 0 to 68718592 [ 3.035955] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.038141] vdb: detected capacity change from 0 to 1073741824 [ 3.049547] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.051603] vdc: detected capacity change from 0 to 2621440000 [ 3.066320] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.069291] vdd: detected capacity change from 0 to 2621440000 [ 3.088652] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.091631] vde: detected capacity change from 0 to 4294967296 [ 3.103738] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.105812] vdf: detected capacity change from 0 to 4294967296 [ 3.112579] libphy: Fixed MDIO Bus: probed [ 3.123319] usbcore: registered new interface driver usbserial_generic [ 3.125126] usbserial: USB Serial support registered for generic [ 3.127133] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.130897] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.132472] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.134431] mousedev: PS/2 mouse device common for all mice [ 3.137785] rtc_cmos 00:05: RTC can wake from S4 [ 3.140844] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.146970] rtc_cmos 00:05: registered as rtc0 [ 3.150727] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.150797] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.152717] intel_pstate: CPU model not supported [ 3.159431] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.165082] hid: raw HID events driver (C) Jiri Kosina [ 3.166732] usbcore: registered new interface driver usbhid [ 3.168372] usbhid: USB HID core driver [ 3.169564] drop_monitor: Initializing network drop monitor service [ 3.171269] Initializing XFRM netlink socket [ 3.172809] NET: Registered protocol family 10 [ 3.175223] Segment Routing with IPv6 [ 3.176377] NET: Registered protocol family 17 [ 3.178245] mpls_gso: MPLS GSO support [ 3.184793] RAS: Correctable Errors collector initialized. [ 3.187391] AVX version of gcm_enc/dec engaged. [ 3.189136] AES CTR mode by8 optimization enabled [ 3.262978] sched_clock: Marking stable (3262954647, 0)->(4217469326, -954514679) [ 3.268321] registered taskstats version 1 [ 3.270729] Loading compiled-in X.509 certificates [ 3.273919] zswap: loaded using pool lzo/zbud [ 3.298338] Key type big_key registered [ 3.311380] Key type encrypted registered [ 3.313641] ima: No TPM chip found, activating TPM-bypass! [ 3.316750] ima: Allocated hash algorithm: sha1 [ 3.319433] ima: No architecture policies found [ 3.321969] evm: Initialising EVM extended attributes: [ 3.324846] evm: security.selinux [ 3.326677] evm: security.ima [ 3.328239] evm: security.capability [ 3.330300] evm: HMAC attrs: 0x1 [ 3.333192] rtc_cmos 00:05: setting system clock to 2026-01-30 06:01:11 UTC (1769752871) [ 3.340675] debug: unmapping init [mem 0xffffffffa8203000-0xffffffffa83fffff] [ 3.345367] debug: unmapping init [mem 0xffffffffa6f82000-0xffffffffa7258fff] [ 3.355075] Write protecting the kernel read-only data: 28672k [ 3.359510] debug: unmapping init [mem 0xffffffffa5603000-0xffffffffa57fffff] [ 3.363485] debug: unmapping init [mem 0xffffffffa5f14000-0xffffffffa5ffffff] [ 3.397994] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.409437] systemd[1]: Detected virtualization kvm. [ 3.410967] systemd[1]: Detected architecture x86-64. [ 3.412353] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.440863] systemd[1]: No hostname configured. [ 3.442707] systemd[1]: Set hostname to . [ 3.445102] random: systemd: uninitialized urandom read (16 bytes read) [ 3.447596] systemd[1]: Initializing machine ID from random generator. [ 3.504397] random: ln: uninitialized urandom read (6 bytes read) [ 3.598727] random: systemd: uninitialized urandom read (16 bytes read) [ 3.602469] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.606641] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.611068] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.190796] device-mapper: uevent: version 1.0.3 [ 4.193068] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.835769] virtio_net virtio0 ens2: renamed from eth0 [ 4.838738] random: fast init done [ 4.854230] scsi host0: ata_piix [ 4.860877] scsi host1: ata_piix [ 4.862612] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.865862] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.240646] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.854165] random: crng init done [ 9.855445] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.208714] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.342790] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.596295] SELinux: Disabled at runtime. [ 11.655424] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.666283] systemd[1]: Detected virtualization kvm. [ 11.668420] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.088438] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.091647] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.095638] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.102522] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.106124] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.111560] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.119606] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ 12.243774] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.566813] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.889387] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.935457] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.050239] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.068171] EDAC sbridge: Ver: 1.1.2 [ 14.750393] Key type dns_resolver registered [ 15.089916] NFS: Registering the id_resolver key type [ 15.091983] Key type id_resolver registered [ 15.094193] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg308-server login: [ 45.463228] libcfs: loading out-of-tree module taints kernel. [ 45.481515] Key type ._llcrypt registered [ 45.483042] Key type .llcrypt registered [ 45.586075] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 64.309637] hrtimer: interrupt took 33053746 ns [ 66.099794] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 68.363772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.413405] alg: No test for adler32 (adler32-zlib) [ 69.910600] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 70.893577] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 72.711173] Key type lgssc registered [ 75.440255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.613708] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.307976] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 145.899996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.946487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.309029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.377206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.536743] Lustre: lustre-MDT0000: new disk, initializing [ 147.677655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.703164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.556266] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.662784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 166.791263] Lustre: 6519:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.819407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.829867] Lustre: Skipped 1 previous similar message [ 166.916811] Lustre: lustre-MDT0001: new disk, initializing [ 166.980655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.013372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.028310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 171.828170] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.832596] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.246671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 186.684306] Lustre: lustre-OST0000: new disk, initializing [ 186.693144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.819576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.273847] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.151486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.158754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.273555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.890918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 207.062390] Lustre: lustre-OST0001: new disk, initializing [ 207.067537] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.259078] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.339831] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.081880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.105300] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 215.237620] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.669930] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.722452] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.478758] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing check_logdir /tmp/testlogs/ [ 249.323163] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing yml_node [ 253.729715] Lustre: DEBUG MARKER: Client: 2.17.50.72 [ 256.134134] Lustre: DEBUG MARKER: MDS: 2.17.50.72 [ 258.525686] Lustre: DEBUG MARKER: OSS: 2.17.50.72 [ 260.128865] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 30 01:05:27 EST 2026 [ 277.861727] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 279.393291] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.442183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.443494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.454503] Lustre: Skipped 3 previous similar messages [ 297.463613] Lustre: Skipped 2 previous similar messages [ 301.638769] LustreError: 12213:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 301.873596] Lustre: server umount lustre-MDT0000 complete [ 305.552368] LustreError: 6512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753173 with bad export cookie 5781442862962753395 [ 305.554333] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.563357] LustreError: 6512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 307.682630] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.685485] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.686930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.686939] Lustre: Skipped 1 previous similar message [ 307.705567] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 307.735522] Lustre: Skipped 1 previous similar message [ 310.687932] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.687971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.722445] Lustre: Skipped 1 previous similar message [ 311.798983] LustreError: 12414:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 311.810638] LustreError: 12414:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 312.080490] Lustre: server umount lustre-MDT0001 complete [ 316.032096] LustreError: 12617:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 316.038346] LustreError: 12617:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 316.106924] Lustre: server umount lustre-OST0000 complete [ 320.198495] LustreError: 12817:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 320.202945] LustreError: 12817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 320.339419] Lustre: server umount lustre-OST0001 complete [ 326.915786] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 334.941500] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 372.152189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.455893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 372.527242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 372.640289] Lustre: lustre-MDT0000: new disk, initializing [ 372.750365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 372.779749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 377.070637] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 387.069936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 387.224296] Lustre: 14991:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 387.304092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 387.307812] Lustre: Skipped 1 previous similar message [ 387.382362] Lustre: lustre-MDT0001: new disk, initializing [ 387.446790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 387.499298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 387.521152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.242432] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 397.810173] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 400.857391] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 406.366519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 406.618501] Lustre: lustre-OST0000: new disk, initializing [ 406.625597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 406.735495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.250404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 408.258829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 408.387053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 411.660351] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 419.129282] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 424.212613] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 424.398425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 427.707752] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 428.094504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 434.150744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.153236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.167689] LustreError: Skipped 1 previous similar message [ 434.192725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.273484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.280141] Lustre: Skipped 1 previous similar message [ 444.383178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.426145] LustreError: 17926:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 444.428856] LustreError: 17926:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 444.544155] Lustre: server umount lustre-OST0000 complete [ 448.992622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 449.010704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.042696] Lustre: Skipped 1 previous similar message [ 449.077621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.079799] Lustre: Skipped 3 previous similar messages [ 463.327659] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 463.434727] LustreError: 18129:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 463.446624] LustreError: 18129:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 463.728801] Lustre: server umount lustre-MDT0000 complete [ 464.868368] LustreError: 16121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.892212] LustreError: 16121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 467.564458] LustreError: 14985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753335 with bad export cookie 5781442862962754935 [ 467.570369] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 467.575440] LustreError: 14985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 468.140200] Lustre: server umount lustre-MDT0001 complete [ 474.449188] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:09:01 (1769753341) [ 494.190928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.705054] LustreError: 19466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.786656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.513304] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 500.195047] LustreError: 19466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.517415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 511.433247] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 517.299184] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 520.990371] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 522.816427] Lustre: Failing over lustre-MDT0000 [ 522.985594] LustreError: 20807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 522.995379] LustreError: 20807:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 523.117488] Lustre: server umount lustre-MDT0000 complete [ 524.257689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.259164] LustreError: 19466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.284216] Lustre: Skipped 2 previous similar messages [ 524.312384] LustreError: 19466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 540.640237] Lustre: 3660:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769753392/real 1769753392] req@ffff8af33ee7e680 x1855720457922432/t0(0) o400->MGC192.168.203.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769753408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 540.692284] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.739174] LustreError: 19466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.782142] LustreError: 19466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 543.636153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.211389] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 551.220942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 551.262598] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 555.656688] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 563.084865] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 570.868303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.886263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 570.890470] Lustre: Skipped 6 previous similar messages [ 582.111526] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 582.223653] LustreError: 21785:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 582.236562] LustreError: 21785:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 582.498964] Lustre: server umount lustre-MDT0000 complete [ 586.651908] LustreError: 19446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753454 with bad export cookie 5781442862962756349 [ 586.659862] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.665449] LustreError: 19446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.147729] Lustre: server umount lustre-MDT0001 complete [ 595.877888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.465727] LustreError: 22465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.492107] LustreError: 22465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 596.541562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.547759] Lustre: Skipped 2 previous similar messages [ 601.630588] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 610.190358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 615.011782] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 621.099024] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 624.877612] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 626.835130] Lustre: Failing over lustre-MDT0000 [ 627.045374] Lustre: server umount lustre-MDT0000 complete [ 631.264453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.273477] Lustre: Skipped 2 previous similar messages [ 645.270140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.422240] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.606649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 645.614085] Lustre: Skipped 6 previous similar messages [ 645.745524] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 650.245209] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 650.720687] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 650.730130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 650.744951] Lustre: Skipped 1 previous similar message [ 650.773515] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 658.000874] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 665.057314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.319313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 676.396252] LustreError: 24782:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 676.401293] LustreError: 24782:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 676.541731] Lustre: server umount lustre-MDT0000 complete [ 679.948076] LustreError: 22444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753548 with bad export cookie 5781442862962757287 [ 679.949490] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.961563] LustreError: 22444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 680.388915] Lustre: server umount lustre-MDT0001 complete [ 688.091778] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:12:35 (1769753555) [ 703.958995] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 712.728664] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 752.713507] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 761.093200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.575991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 761.695408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 761.874550] Lustre: lustre-MDT0000: new disk, initializing [ 761.986696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.989760] Lustre: Skipped 2 previous similar messages [ 762.004079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 765.001165] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 777.359886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 777.488403] Lustre: 28446:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 777.746928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 783.105361] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 791.808618] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 796.441388] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 806.441575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 808.151875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 808.230310] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 808.390443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 808.444033] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 815.147723] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 822.913646] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 828.501992] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 844.271479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 844.274839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.283772] LustreError: Skipped 1 previous similar message [ 844.287989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 844.299407] Lustre: Skipped 2 previous similar messages [ 844.314673] Lustre: Skipped 6 previous similar messages [ 850.366088] LustreError: 31099:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 850.385988] LustreError: 31099:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 850.594560] Lustre: server umount lustre-OST0000 complete [ 859.618323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 863.785432] Lustre: server umount lustre-MDT0000 complete [ 864.737607] LustreError: 29294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.772625] LustreError: 29294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 868.642938] LustreError: 28439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753736 with bad export cookie 5781442862962757693 [ 868.654147] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 868.667509] LustreError: 28439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 869.442242] Lustre: server umount lustre-MDT0001 complete [ 881.321904] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 884.491085] Key type lgssc unregistered [ 884.920149] LNet: 32037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.931118] LNetError: 32037:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.960672] LNet: Removed LNI 192.168.203.108@tcp [ 886.374499] Key type .llcrypt unregistered [ 886.380248] Key type ._llcrypt unregistered [ 912.500287] Key type ._llcrypt registered [ 912.502660] Key type .llcrypt registered [ 912.600799] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 913.984690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.036133] alg: No test for adler32 (adler32-zlib) [ 914.990403] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 915.263782] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 916.975229] Key type lgssc registered [ 918.328355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 927.126694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 927.162145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.692519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.329501] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 937.958514] LustreError: 33434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.978966] LustreError: 33434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 943.073244] LustreError: 33434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 943.083137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 943.297077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 949.023861] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 955.528698] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 960.450603] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 966.781018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 967.091343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.125628] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 973.292846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 973.397752] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 979.023583] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 995.236845] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:17:41 (1769753861) [ 997.418366] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 999.730323] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1007.239361] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:17:54 (1769753874) [ 1015.489585] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:18:02 (1769753882) [ 1016.438201] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1025.548154] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1025.557580] Lustre: Skipped 2 previous similar messages [ 1027.950657] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:18:14 (1769753894) [ 1030.260333] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1030.268954] Lustre: Skipped 2 previous similar messages [ 1040.211885] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1040.219116] Lustre: Skipped 4 previous similar messages [ 1055.989967] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:18:42 (1769753922) [ 1059.250408] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1063.525682] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1073.538992] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:19:00 (1769753940) [ 1075.723327] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1075.728255] Lustre: Skipped 6 previous similar messages [ 1089.542445] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:19:16 (1769753956) [ 1106.411360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.415526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1106.447342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1111.521453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1111.534960] Lustre: Skipped 2 previous similar messages [ 1111.640121] LustreError: 39999:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1111.844925] Lustre: server umount lustre-OST0000 complete [ 1116.641150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.659394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.665052] Lustre: Skipped 2 previous similar messages [ 1116.682754] Lustre: Skipped 1 previous similar message [ 1121.765548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.772549] Lustre: Skipped 1 previous similar message [ 1122.584177] LustreError: 40201:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1122.599056] LustreError: 40201:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1122.817489] Lustre: server umount lustre-MDT0000 complete [ 1126.886129] LustreError: 33435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.910882] LustreError: 33435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1127.325461] LustreError: 33416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769753995 with bad export cookie 17686973940977674153 [ 1127.331274] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.338148] LustreError: 33416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1127.571278] LustreError: 40403:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1127.579237] LustreError: 40403:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1127.916152] Lustre: server umount lustre-MDT0001 complete [ 1140.542546] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1144.160656] Key type lgssc unregistered [ 1144.492900] LNet: 40934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1144.498719] LNetError: 40934:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.513588] LNet: Removed LNI 192.168.203.108@tcp [ 1146.669836] Key type .llcrypt unregistered [ 1146.671751] Key type ._llcrypt unregistered [ 1170.233705] Key type ._llcrypt registered [ 1170.236731] Key type .llcrypt registered [ 1170.417270] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 1187.744477] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1189.183933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1189.276468] alg: No test for adler32 (adler32-zlib) [ 1190.435261] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 1190.738266] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1192.415159] Key type lgssc registered [ 1193.847111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1232.895815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1232.974384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.421656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1234.502237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1234.648405] Lustre: lustre-MDT0000: new disk, initializing [ 1234.770104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.827059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1240.790112] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1254.082511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1254.238128] Lustre: 44054:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1254.313859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1254.317930] Lustre: Skipped 1 previous similar message [ 1254.402713] Lustre: lustre-MDT0001: new disk, initializing [ 1254.488777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1254.548663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1254.556731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1259.965110] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1268.528816] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1273.398884] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1280.458401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1280.765826] Lustre: lustre-OST0000: new disk, initializing [ 1280.773694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1280.878067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.759394] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1288.807454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1288.826783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1288.992286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1295.581049] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1300.207220] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1300.443457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1304.972363] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1305.168422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1309.664875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1309.670500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.679854] LustreError: Skipped 1 previous similar message [ 1309.680823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1309.711098] Lustre: Skipped 1 previous similar message [ 1313.222047] LustreError: 46987:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1313.692316] Lustre: server umount lustre-OST0000 complete [ 1319.911598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.932351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.940653] Lustre: Skipped 2 previous similar messages [ 1324.394821] LustreError: 47189:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1324.399806] LustreError: 47189:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1324.847941] Lustre: server umount lustre-MDT0000 complete [ 1328.641470] LustreError: 44833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754196 with bad export cookie 958412460280464223 [ 1328.644110] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.652977] LustreError: 44833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1328.800667] LustreError: 47390:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1328.808554] LustreError: 47390:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1329.222166] Lustre: server umount lustre-MDT0001 complete [ 1340.629403] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:23:27 (1769754207) [ 1347.971949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.697443] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.746835] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1348.793593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.214854] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.550714] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1358.304627] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.468027] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.948358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1366.566607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1371.510553] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1377.691747] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1381.041541] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1388.485607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1389.306236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.123904] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1404.354440] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1408.586390] LustreError: 50301:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1409.695210] Lustre: Modifying parameter general.timeout=1 in log params [ 1410.745480] Lustre: Modifying parameter general.timeout=2 in log params [ 1413.134206] Lustre: Modifying parameter general.timeout=4 in log params [ 1413.140670] Lustre: Skipped 1 previous similar message [ 1417.871920] Lustre: Modifying parameter general.timeout=8 in log params [ 1417.879165] Lustre: Skipped 3 previous similar messages [ 1426.215748] Lustre: Modifying parameter general.timeout=17 in log params [ 1426.220715] Lustre: Skipped 8 previous similar messages [ 1442.652556] Lustre: Modifying parameter general.timeout=35 in log params [ 1442.657855] Lustre: Skipped 17 previous similar messages [ 1474.977433] Lustre: Modifying parameter general.timeout=72 in log params [ 1474.982450] Lustre: Skipped 36 previous similar messages [ 1539.518434] Lustre: Disabling parameter general.timeout= in log params [ 1539.522590] Lustre: Skipped 57 previous similar messages [ 1543.932355] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:26:50 (1769754410) [ 1547.111887] Lustre: 57241:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1549.575974] Lustre: 57350:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1553.516387] Lustre: 57546:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1553.530447] Lustre: 57546:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1557.544190] Lustre: 57759:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1557.553411] Lustre: 57759:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1583.968365] Lustre: 59116:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1583.983162] LustreError: 59116:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1587.673606] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:27:35 (1769754455) [ 1602.018823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1602.032646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.051856] Lustre: Skipped 1 previous similar message [ 1602.057347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1606.140168] LustreError: 59818:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1606.155313] LustreError: 59818:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1606.298321] Lustre: server umount lustre-MDT0000 complete [ 1608.185179] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1608.230683] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1609.350016] LustreError: 48090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754477 with bad export cookie 958412460280465105 [ 1609.351666] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1609.367479] LustreError: 48090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1609.711053] Lustre: server umount lustre-MDT0001 complete [ 1623.358207] LustreError: 60220:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1623.363236] LustreError: 60220:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1623.477920] Lustre: server umount lustre-OST0000 complete [ 1651.621036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.050140] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1652.275823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.555360] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1666.505869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1666.658453] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1666.712028] Lustre: 61649:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1671.929727] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1678.223220] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1682.590825] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1689.212515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1689.382772] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1689.646691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.650199] Lustre: Skipped 1 previous similar message [ 1695.313135] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1701.963622] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1704.505613] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1704.518293] Lustre: Skipped 23 previous similar messages [ 1704.594781] Lustre: 63876:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1714.658880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.661354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.664665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.664672] Lustre: Skipped 3 previous similar messages [ 1714.673539] LustreError: Skipped 1 previous similar message [ 1714.727678] Lustre: Skipped 3 previous similar messages [ 1719.319383] LustreError: 64133:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1719.323564] LustreError: 64133:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1719.682539] Lustre: server umount lustre-OST0000 complete [ 1724.896206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.899944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1724.909889] Lustre: Skipped 1 previous similar message [ 1724.918356] Lustre: Skipped 2 previous similar messages [ 1730.979651] Lustre: server umount lustre-MDT0000 complete [ 1734.117200] LustreError: 61660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.132044] LustreError: 61660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1734.744943] LustreError: 62373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754602 with bad export cookie 958412460280468080 [ 1734.756654] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1734.763397] LustreError: 62373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1735.063938] Lustre: server umount lustre-MDT0001 complete [ 1746.541795] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1749.294585] Key type lgssc unregistered [ 1749.597737] LNet: 65068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.602708] LNetError: 65068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.632709] LNet: Removed LNI 192.168.203.108@tcp [ 1750.619803] Key type .llcrypt unregistered [ 1750.622024] Key type ._llcrypt unregistered [ 1776.271475] Key type ._llcrypt registered [ 1776.275943] Key type .llcrypt registered [ 1776.409827] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1777.799712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1777.860058] alg: No test for adler32 (adler32-zlib) [ 1779.119289] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 1779.488937] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1781.215369] Key type lgssc registered [ 1782.449729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1791.352878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1791.377817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.036860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1798.158380] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1803.233576] LustreError: 66465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1803.249465] LustreError: 66465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1808.030861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1808.537576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1812.986909] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1819.343938] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1823.189815] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1829.625146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1830.223783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.600785] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1843.113180] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1852.110795] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1864.161362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.165222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.188059] Lustre: Skipped 2 previous similar messages [ 1864.209577] Lustre: Skipped 2 previous similar messages [ 1867.082581] LustreError: 69167:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1867.228641] Lustre: server umount lustre-MDT0000 complete [ 1869.280530] LustreError: 67179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.323849] LustreError: 67179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1871.346157] LustreError: 66447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754739 with bad export cookie 7609398794198409441 [ 1871.368362] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.372853] LustreError: 66447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1871.545231] LustreError: 69371:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1871.551303] LustreError: 69371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1871.867736] Lustre: server umount lustre-MDT0001 complete [ 1886.640404] LustreError: 69572:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1886.648257] LustreError: 69572:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1886.940476] Lustre: server umount lustre-OST0000 complete [ 1915.366227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.701198] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1915.738423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1916.063733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.374702] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1930.322569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1930.445863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1930.497594] Lustre: 71002:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1930.616989] LustreError: 71013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.193403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.905030] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1942.009339] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1946.663163] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1954.524823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1954.664913] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1955.095227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.460215] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1967.125820] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1968.723862] Lustre: 73236:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1971.869793] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1981.925346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1981.938963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.967253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1982.435565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1984.840247] LustreError: 73499:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1984.851912] LustreError: 73499:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1985.142159] Lustre: server umount lustre-OST0000 complete [ 1992.160375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1992.164463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.170294] Lustre: Skipped 1 previous similar message [ 1992.174970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.177376] Lustre: Skipped 2 previous similar messages [ 1995.744988] LustreError: 73700:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1995.748749] LustreError: 73700:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1995.985752] Lustre: server umount lustre-MDT0000 complete [ 1997.791730] LustreError: 73363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.825520] LustreError: 73363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2000.210975] LustreError: 70993:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754868 with bad export cookie 7609398794198410274 [ 2000.216100] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.227857] LustreError: 70993:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2000.733540] Lustre: server umount lustre-MDT0001 complete [ 2012.800737] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2016.815748] Key type lgssc unregistered [ 2017.356801] LNet: 74434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.379898] LNetError: 74434:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.417911] LNet: Removed LNI 192.168.203.108@tcp [ 2019.759387] Key type .llcrypt unregistered [ 2019.763351] Key type ._llcrypt unregistered [ 2048.310425] Key type ._llcrypt registered [ 2048.316049] Key type .llcrypt registered [ 2048.526921] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2050.112102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.208083] alg: No test for adler32 (adler32-zlib) [ 2051.321159] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 2051.565532] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2053.273598] Key type lgssc registered [ 2054.590992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2062.552709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2062.590719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.309775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.988534] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2074.595477] LustreError: 75831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.606961] LustreError: 75831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2078.134761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2078.582314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.919613] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2091.686366] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2096.951160] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2104.175859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2104.554365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.618875] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2116.327146] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2125.848733] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2126.753970] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2128.428376] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2128.446104] Lustre: Skipped 1 previous similar message [ 2130.673312] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2130.677832] Lustre: Skipped 1 previous similar message [ 2135.479591] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2135.489620] Lustre: Skipped 3 previous similar messages [ 2143.683425] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2143.687364] Lustre: Skipped 8 previous similar messages [ 2160.407706] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2160.412666] Lustre: Skipped 14 previous similar messages [ 2192.646151] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2192.648844] Lustre: Skipped 34 previous similar messages [ 2229.761563] Lustre: 83602:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2232.770443] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:38:19 (1769755099) [ 2234.518566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2236.492450] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:38:23 (1769755103) [ 2241.006615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2241.013567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.032880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2246.112954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2246.115511] Lustre: Skipped 1 previous similar message [ 2251.232755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2251.246296] Lustre: Skipped 1 previous similar message [ 2255.328322] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2255.393102] LustreError: 83898:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2255.641511] Lustre: server umount lustre-OST0000 complete [ 2261.480764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.493648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2261.516296] Lustre: Skipped 2 previous similar messages [ 2261.541351] Lustre: Skipped 1 previous similar message [ 2266.593722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.613143] Lustre: Skipped 1 previous similar message [ 2266.963097] LustreError: 84101:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2266.987675] LustreError: 84101:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2267.255801] Lustre: server umount lustre-MDT0000 complete [ 2271.712188] LustreError: 75832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.742971] LustreError: 75832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2272.330815] LustreError: 77457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755140 with bad export cookie 2456408851009668814 [ 2272.335152] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2272.344867] LustreError: 77457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2272.519189] LustreError: 84303:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2272.526870] LustreError: 84303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2272.726994] Lustre: server umount lustre-MDT0001 complete [ 2285.255789] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2288.219758] Key type lgssc unregistered [ 2288.535865] LNet: 84835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.545423] LNetError: 84835:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.560856] LNet: Removed LNI 192.168.203.108@tcp [ 2289.431449] Key type .llcrypt unregistered [ 2289.433909] Key type ._llcrypt unregistered [ 2294.761626] Key type ._llcrypt registered [ 2294.764844] Key type .llcrypt registered [ 2294.911340] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2299.471433] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules [ 2300.500175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2300.533323] alg: No test for adler32 (adler32-zlib) [ 2301.795862] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 2302.014811] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2303.698074] Key type lgssc registered [ 2305.157032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2306.496298] LustreError: 86002:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 2324.808550] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2328.789309] LustreError: 86002:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2334.979733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2334.992492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2336.524496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.967568] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2345.955604] LustreError: 87008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.978721] LustreError: 87008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2351.079197] LustreError: 87008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.835640] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:40:18 (1769755218) [ 2356.198840] LustreError: 87008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.534678] LustreError: 87040:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2357.552225] LustreError: 87040:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2357.568718] LustreError: 87040:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2357.746791] LustreError: 87813:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2357.950290] Lustre: server umount lustre-MDT0000 complete [ 2374.367598] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2377.874897] Key type lgssc unregistered [ 2378.097061] LNet: 88498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.105820] LNetError: 88498:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.121395] LNet: Removed LNI 192.168.203.108@tcp [ 2379.154694] Key type .llcrypt unregistered [ 2379.157267] Key type ._llcrypt unregistered [ 2398.599932] Key type ._llcrypt registered [ 2398.602519] Key type .llcrypt registered [ 2398.735708] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2399.832148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.068557] alg: No test for adler32 (adler32-zlib) [ 2401.226786] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 2401.484742] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2403.167266] Key type lgssc registered [ 2404.521744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2413.688838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2413.725420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.382285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.481407] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2425.832763] LustreError: 89699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.858226] LustreError: 89699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2428.842530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2429.638430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2433.799676] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2439.162133] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2443.305953] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2449.401158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2449.934545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.482423] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2460.953829] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2508.137282] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:42:55 (1769755375) [ 2510.167845] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2512.480236] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:42:59 (1769755379) [ 2521.568627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2521.576790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.587250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2521.613601] Lustre: Skipped 2 previous similar messages [ 2523.927114] LustreError: 92603:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2524.140982] Lustre: server umount lustre-MDT0000 complete [ 2526.689336] LustreError: 89695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.702243] LustreError: 89695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2527.974639] LustreError: 89679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755396 with bad export cookie 12939734749941684433 [ 2527.979132] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.989539] LustreError: 89679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2528.146968] LustreError: 92805:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2528.151947] LustreError: 92805:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2528.320712] Lustre: server umount lustre-MDT0001 complete [ 2543.188275] LustreError: 93007:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2543.200356] LustreError: 93007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2543.608782] Lustre: server umount lustre-OST0000 complete [ 2556.373718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.975299] LustreError: 93636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.187644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.006239] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2562.531078] LustreError: 93635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.552187] LustreError: 93635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2567.651598] LustreError: 93635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.955829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2572.061606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2576.227534] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2581.053688] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2584.938486] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2599.063794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2599.364709] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2599.373860] LustreError: 93624:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2599.422376] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2599.449861] LustreError: 95491:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2599.453164] LustreError: 95491:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2599.472169] LustreError: 95491:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2599.557320] Lustre: server umount lustre-OST0000 complete [ 2599.560142] LustreError: 95491:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2609.792585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2609.945902] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2609.959265] LustreError: 93623:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2609.980344] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2609.991906] LustreError: 95957:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2609.996623] LustreError: 95957:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2610.005744] LustreError: 95957:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2610.088662] Lustre: server umount lustre-OST0000 complete [ 2610.090694] LustreError: 95957:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2620.539692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2620.904618] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2620.913125] Lustre: Found index 0 for lustre-OST0000, updating log [ 2620.917682] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2621.081701] Lustre: lustre-OST0000: new disk, initializing [ 2621.103063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2621.736812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.919624] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2632.776149] Lustre: Failing over lustre-OST0000 [ 2632.848941] LustreError: 97040:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2632.853566] LustreError: 97040:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2632.986547] Lustre: server umount lustre-OST0000 complete [ 2637.286093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.288029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.307435] Lustre: Skipped 1 previous similar message [ 2637.325233] Lustre: Skipped 3 previous similar messages [ 2642.401577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.416293] Lustre: Skipped 1 previous similar message [ 2648.555860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.561222] Lustre: Skipped 3 previous similar messages [ 2651.108248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2651.210969] LustreError: 97242:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2651.217461] LustreError: 97242:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2651.415530] Lustre: server umount lustre-MDT0000 complete [ 2653.665273] LustreError: 93636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.699463] LustreError: 93636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2655.146821] LustreError: 93616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755523 with bad export cookie 12939734749941687177 [ 2655.150325] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2655.161532] LustreError: 93616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2655.631661] Lustre: server umount lustre-MDT0001 complete [ 2664.240744] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:45:31 (1769755531) [ 2684.017877] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2693.583336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.928214] LustreError: 98811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.985089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.341293] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2706.003473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2710.876587] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2713.633613] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2720.368244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2721.292653] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2728.272663] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2731.500844] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2731.512190] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2731.524730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2736.041916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2736.145802] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2736.220164] Lustre: lustre-OST0001: new disk, initializing [ 2736.223772] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2736.462633] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2736.483206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2736.499259] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2736.505437] Lustre: Skipped 2 previous similar messages [ 2739.564430] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:01 [ 2739.590247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2739.607355] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2739.607704] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2739.623331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2742.795933] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2745.977092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2745.984802] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2746.108811] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2750.783144] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2758.529280] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2768.368050] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2768.382416] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.398032] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2771.951303] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.965866] Lustre: Skipped 1 previous similar message [ 2776.501891] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.8@tcp (stopping) [ 2776.511793] Lustre: Skipped 4 previous similar messages [ 2782.175372] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2782.280550] LustreError: 102049:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2782.293351] LustreError: 102049:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2782.428674] Lustre: server umount lustre-MDT0001 complete [ 2786.746454] LustreError: 98806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.757303] LustreError: 98806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2791.513332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2791.614980] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2791.623377] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2791.682528] Lustre: 98800:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2793.149684] LustreError: 102642:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2793.164555] LustreError: 102642:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2793.170047] LustreError: 98802:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2793.345076] LustreError: 102656:0:(obd_config.c:2075:class_config_llog_handler()) MGC192.168.203.108@tcp: cfg command failed: rc = -17 [ 2793.351044] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2793.351044] [ 2796.692328] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2797.567024] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2805.018887] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 01:47:52 (1769755672) [ 2822.285255] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2838.867834] Lustre: 104651:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2859.517244] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2862.725958] Lustre: 105784:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2945.506940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2945.515396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.537248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2945.551233] Lustre: Skipped 7 previous similar messages [ 2948.685485] LustreError: 106237:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2948.695796] LustreError: 106237:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2948.967145] Lustre: server umount lustre-MDT0000 complete [ 2951.137228] LustreError: 98805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.157455] LustreError: 98805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2952.375858] LustreError: 99918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755820 with bad export cookie 12939734749941688171 [ 2952.378916] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2952.392781] LustreError: 99918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2952.727785] Lustre: server umount lustre-MDT0001 complete [ 2966.036451] Lustre: server umount lustre-OST0000 complete [ 2980.557506] Lustre: server umount lustre-OST0001 complete [ 2988.180700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2999.854964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3007.128982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3017.412798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3038.205970] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3046.953479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.974695] Lustre: lustre-MDT0000: reset Object Index mappings [ 3047.586811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.591758] Lustre: Skipped 1 previous similar message [ 3052.004679] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3060.793059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3060.809503] Lustre: lustre-MDT0001: reset Object Index mappings [ 3066.707587] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3069.585250] Lustre: 110830:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3075.728509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3083.825934] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3086.715532] LustreError: 111186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.742344] LustreError: 111186:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3086.756139] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3091.930938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3092.053842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3094.181169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3094.181414] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3099.298573] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3106.266158] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3109.343679] Lustre: 112679:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3158.825881] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:53:46 (1769756026) [ 3188.193382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.209749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.220963] Lustre: Skipped 6 previous similar messages [ 3188.224050] Lustre: Skipped 6 previous similar messages [ 3192.462418] LustreError: 113030:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3192.470890] LustreError: 113030:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3192.852718] Lustre: server umount lustre-MDT0000 complete [ 3197.265419] LustreError: 109697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756065 with bad export cookie 12939734749941735274 [ 3197.278541] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.837155] Lustre: server umount lustre-MDT0001 complete [ 3208.179733] LustreError: 113434:0:(ldlm_resource.c:1170:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x63:0x0].0x0 (ffff8af20d878b00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3214.504231] Lustre: server umount lustre-OST0000 complete [ 3224.725825] Lustre: server umount lustre-OST0001 complete [ 3231.828773] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3241.421511] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3274.529209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.744285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3274.765625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3274.834563] Lustre: lustre-MDT0000: new disk, initializing [ 3274.926717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.931660] Lustre: Skipped 3 previous similar messages [ 3274.948340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3278.690078] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3284.723580] LustreError: 116374:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3284.730661] LustreError: 116374:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3284.918904] Lustre: server umount lustre-MDT0000 complete [ 3293.100581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.501270] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3293.523449] Lustre: 116890:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3293.544914] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3297.454949] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3302.210363] Lustre: server umount lustre-MDT0000 complete [ 3310.084673] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:56:17 (1769756177) [ 3311.383602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3312.958299] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:56:20 (1769756180) [ 3327.912351] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3336.889716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.347609] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3349.618203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3349.830989] Lustre: 118903:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3349.920202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3349.925166] Lustre: Skipped 1 previous similar message [ 3350.001634] Lustre: lustre-MDT0001: new disk, initializing [ 3350.259113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3350.274847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3355.280180] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3359.833979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3366.174122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3366.557977] Lustre: lustre-OST0000: new disk, initializing [ 3366.563597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3367.796942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3367.801736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3367.918988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3372.122027] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3382.147906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3382.310860] Lustre: lustre-OST0001: new disk, initializing [ 3382.315475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3383.695067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3383.714303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3383.837555] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3389.142985] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3399.737397] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3403.703165] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3761.406831] Lustre: 127704:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3761.423906] Lustre: 127704:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3761.433163] Lustre: 127704:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3761.447610] Lustre: 127704:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3761.456027] Lustre: 127704:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3761.474405] Lustre: 127704:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4033.908763] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:08:20 (1769756900) [ 4051.442887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4051.457658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.472051] Lustre: Skipped 2 previous similar messages [ 4051.476562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.480895] Lustre: Skipped 5 previous similar messages [ 4054.848605] LustreError: 127942:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4054.854079] LustreError: 127942:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4055.288864] Lustre: server umount lustre-MDT0000 complete [ 4057.574597] LustreError: 122100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.600737] LustreError: 122100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 4059.568254] LustreError: 118895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756927 with bad export cookie 12939734749941801144 [ 4059.581316] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.587131] LustreError: 118895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4062.689187] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.697316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4062.705540] Lustre: Skipped 4 previous similar messages [ 4062.730952] Lustre: Skipped 4 previous similar messages [ 4066.198941] Lustre: server umount lustre-MDT0001 complete [ 4076.158592] LustreError: 128347:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4076.170012] LustreError: 128347:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4076.432548] Lustre: server umount lustre-OST0000 complete [ 4086.867402] Lustre: server umount lustre-OST0001 complete [ 4093.686772] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 4104.664633] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 4142.952713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.338799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4143.397317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4143.533164] Lustre: lustre-MDT0000: new disk, initializing [ 4143.640096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.649312] Lustre: Skipped 5 previous similar messages [ 4143.690239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4147.883696] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4158.900629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4159.008256] Lustre: 130721:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4159.069668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4159.074257] Lustre: Skipped 1 previous similar message [ 4159.169862] Lustre: lustre-MDT0001: new disk, initializing [ 4159.362367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4159.375114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4163.521409] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4170.246382] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4173.899636] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4180.621774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4180.859847] Lustre: lustre-OST0000: new disk, initializing [ 4180.863968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4180.948018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4180.952159] Lustre: Skipped 1 previous similar message [ 4182.062741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4182.076454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4182.235287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4186.689786] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 4192.911184] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4204.801944] Lustre: lustre-MDD0000: changelog on [ 4207.545571] Lustre: lustre-MDD0001: changelog on [ 5599.154882] Lustre: lustre-MDD0001: changelog off [ 5600.329684] Lustre: 130728:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5600.852564] Lustre: lustre-MDD0000: changelog off [ 5605.866651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.872891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5605.883781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5605.883788] Lustre: Skipped 2 previous similar messages [ 5605.904362] LustreError: Skipped 1 previous similar message [ 5610.107266] LustreError: 140455:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5610.114949] LustreError: 140455:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5610.196917] Lustre: server umount lustre-OST0000 complete [ 5613.536409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5613.542622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.553232] Lustre: Skipped 1 previous similar message [ 5613.556081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.561224] Lustre: Skipped 1 previous similar message [ 5616.110268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.744770] LustreError: 140657:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5618.748395] LustreError: 140657:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5618.860987] Lustre: server umount lustre-MDT0000 complete [ 5621.216833] LustreError: 130734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.226954] LustreError: 130734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5621.256843] LustreError: 130715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758489 with bad export cookie 12939734749942617862 [ 5621.257995] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5621.263697] LustreError: 130715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5621.483221] Lustre: server umount lustre-MDT0001 complete [ 5629.333658] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 5631.005504] Key type lgssc unregistered [ 5631.200532] LNet: 141389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5631.204838] LNetError: 141389:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5632.230560] LNet: Removed LNI 192.168.203.108@tcp [ 5632.768140] Key type .llcrypt unregistered [ 5632.771702] Key type ._llcrypt unregistered [ 5644.336687] Key type ._llcrypt registered [ 5644.338091] Key type .llcrypt registered [ 5644.394687] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 5653.666752] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 5654.180060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5654.227390] alg: No test for adler32 (adler32-zlib) [ 5655.152300] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 5655.283764] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 5656.879267] Key type lgssc registered [ 5657.627633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5679.941612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5679.963037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.121457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5681.144229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5681.189399] Lustre: lustre-MDT0000: new disk, initializing [ 5681.228209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.246145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5683.373949] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5689.506360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5689.562420] Lustre: 144532:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5689.605148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5689.609254] Lustre: Skipped 1 previous similar message [ 5689.664779] Lustre: lustre-MDT0001: new disk, initializing [ 5689.730812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5689.760671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5689.767328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5692.053532] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5696.339883] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5698.709437] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5702.779363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5702.920741] Lustre: lustre-OST0000: new disk, initializing [ 5702.923845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5703.008387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.519525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5705.527363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5705.600948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5706.441299] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5711.256817] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5714.673707] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_module obdecho/obdecho [ 5715.856312] LustreError: 147144:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5715.861708] LustreError: 147144:0:(obd_config.c:859:class_setup()) setup ec failed (-22) [ 5716.908358] LustreError: 147240:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5725.664708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5725.670447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.679687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5726.181879] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.185086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5726.188565] Lustre: Skipped 1 previous similar message [ 5726.195292] Lustre: Skipped 1 previous similar message [ 5729.316299] LustreError: 147580:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5729.388373] Lustre: server umount lustre-MDT0000 complete [ 5731.789391] LustreError: 145309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758599 with bad export cookie 4145516226707050079 [ 5731.793803] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5731.804839] LustreError: 145309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5731.914147] LustreError: 147781:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5731.919630] LustreError: 147781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5732.034804] Lustre: server umount lustre-MDT0001 complete [ 5744.634654] LustreError: 147982:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5744.639140] LustreError: 147982:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5744.720904] Lustre: server umount lustre-OST0000 complete [ 5756.460566] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 5761.369355] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 5781.334951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.460460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5781.478540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5781.533123] Lustre: lustre-MDT0000: new disk, initializing [ 5781.568920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.579375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5783.352281] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5788.391500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5788.446982] Lustre: 150914:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5788.478945] LustreError: 150927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.485349] LustreError: 150927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5788.487500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5788.493286] Lustre: Skipped 1 previous similar message [ 5788.536556] Lustre: lustre-MDT0001: new disk, initializing [ 5788.577735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.602707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5788.605870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5790.311431] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5794.231241] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5795.930611] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5799.088674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5799.206688] Lustre: lustre-OST0000: new disk, initializing [ 5799.209270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5799.249863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.807484] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5802.959856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5802.964389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5803.007612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5805.633789] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5807.567644] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5807.653830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5809.548053] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5809.651421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5813.217833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5813.220582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.231046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.238612] Lustre: Skipped 1 previous similar message [ 5813.244261] Lustre: Skipped 1 previous similar message [ 5816.713933] LustreError: 153843:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5816.717147] LustreError: 153843:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5816.771352] Lustre: server umount lustre-OST0000 complete [ 5819.360048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5819.364162] LustreError: Skipped 1 previous similar message [ 5819.366846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.374547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5823.457697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5823.459752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5824.712496] LustreError: 154045:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5824.715506] LustreError: 154045:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5824.754943] Lustre: server umount lustre-MDT0000 complete [ 5826.595124] LustreError: 150907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758694 with bad export cookie 4145516226707051038 [ 5826.596893] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5826.603092] LustreError: 150907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5826.757175] Lustre: server umount lustre-MDT0001 complete [ 5828.695359] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:38:16 (1769758696) [ 5829.514032] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5830.317035] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:38:18 (1769758698) [ 5834.190962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5834.421308] LustreError: 154915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.445664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.223563] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5839.840202] LustreError: 154915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5840.132092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5842.007913] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5844.695694] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5846.481764] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5849.797424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5852.519572] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5855.118621] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5864.418418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.419374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5864.423786] Lustre: Skipped 2 previous similar messages [ 5864.427709] Lustre: Skipped 3 previous similar messages [ 5870.158550] LustreError: 157155:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5870.161172] LustreError: 157155:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5870.199582] Lustre: server umount lustre-MDT0000 complete [ 5871.032541] LustreError: 156645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.130154] LustreError: 154896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758740 with bad export cookie 4145516226707051920 [ 5872.131242] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.138419] LustreError: 154896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5876.148953] LustreError: 156645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.155789] LustreError: 156645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5878.393321] Lustre: server umount lustre-MDT0001 complete [ 5881.809103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5888.976032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5889.193566] LustreError: 158417:0:(llog.c:1643:llog_backup()) MGC192.168.203.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5889.200670] Lustre: 158417:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5895.603331] LustreError: 158424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.609643] LustreError: 158424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5911.995155] LustreError: 158422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5912.003422] LustreError: 158422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5946.807330] LustreError: 158423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.813753] LustreError: 158423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 5950.431403] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5950.434201] Lustre: 158417:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5980.127588] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5980.135277] Lustre: 154907:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5990.888147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5990.892648] Lustre: Skipped 2 previous similar messages [ 5992.933690] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5993.078797] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 5997.962856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.607271] LustreError: 154907:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6012.342977] LustreError: 158422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.351821] LustreError: 158422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 45 previous similar messages [ 6018.531720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6020.770341] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6023.321469] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6024.028985] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6024.168081] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6025.602611] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6028.398822] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6039.079039] LustreError: 159903:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6039.081948] LustreError: 159903:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6039.115697] Lustre: server umount lustre-MDT0000 complete [ 6039.522128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.522330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6039.527494] Lustre: Skipped 2 previous similar messages [ 6041.035365] LustreError: 154895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758909 with bad export cookie 4145516226707058416 [ 6041.038869] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6041.045281] LustreError: 154895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6041.183175] Lustre: server umount lustre-MDT0001 complete [ 6054.668916] Lustre: server umount lustre-OST0000 complete [ 6064.300282] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6068.425194] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6085.411611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.558116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6085.575535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6085.623447] Lustre: lustre-MDT0000: new disk, initializing [ 6085.667382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6085.669929] Lustre: Skipped 1 previous similar message [ 6085.684475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6087.705672] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6093.042438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6093.107223] Lustre: 163236:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6093.174394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6093.178849] Lustre: Skipped 7 previous similar messages [ 6093.248791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6094.972100] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6098.714950] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6100.407070] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6103.359261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6106.052217] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6109.718312] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6111.438467] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6113.282817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6113.344549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6113.591463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6115.317162] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6115.403625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6118.371578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6118.374633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.380116] Lustre: Skipped 2 previous similar messages [ 6122.286104] LustreError: 166178:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6122.289567] LustreError: 166178:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6122.333168] Lustre: server umount lustre-OST0000 complete [ 6128.611465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6128.616244] Lustre: Skipped 3 previous similar messages [ 6130.352197] Lustre: server umount lustre-MDT0000 complete [ 6132.046342] LustreError: 164011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759000 with bad export cookie 4145516226707075216 [ 6132.050143] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6132.051179] LustreError: 164011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6132.193106] Lustre: server umount lustre-MDT0001 complete [ 6136.093713] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:43:24 (1769759004) [ 6139.646418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.804416] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6144.993886] LustreError: 167249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.009963] LustreError: 167249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6146.058724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6148.016176] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6150.827410] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6152.527342] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6155.804827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6158.615698] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6161.009301] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6171.336357] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:43:59 (1769759039) [ 6175.718100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6175.728272] LustreError: Skipped 1 previous similar message [ 6178.837467] Lustre: server umount lustre-OST0000 complete [ 6186.560388] LustreError: 169887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6186.562829] LustreError: 169887:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6186.596557] Lustre: server umount lustre-MDT0000 complete [ 6188.069399] LustreError: 167231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759056 with bad export cookie 4145516226707076098 [ 6188.071555] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.073901] LustreError: 167231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6188.192402] Lustre: server umount lustre-MDT0001 complete [ 6192.306452] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6193.499292] Key type lgssc unregistered [ 6193.662684] LNet: 170622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6193.666799] LNetError: 170622:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6193.676657] LNet: Removed LNI 192.168.203.108@tcp [ 6194.019698] Key type .llcrypt unregistered [ 6194.021232] Key type ._llcrypt unregistered [ 6202.285853] Key type ._llcrypt registered [ 6202.287167] Key type .llcrypt registered [ 6202.329690] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6202.758111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6202.776990] alg: No test for adler32 (adler32-zlib) [ 6203.703076] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 6203.818189] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 6205.423167] Key type lgssc registered [ 6205.928093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6209.847893] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6209.863613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6225.311300] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6230.431155] LustreError: 171821:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 6235.551391] LustreError: 171821:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6235.555694] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6235.557850] LustreError: MGC192.168.203.108@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6235.561528] LustreError: 171821:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6235.564662] LustreError: 171821:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 6235.568079] LustreError: 171821:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 6235.570582] LustreError: 171821:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6235.592747] Lustre: server umount lustre-OST0000 complete [ 6235.594376] LustreError: 171821:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6238.948616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.153605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6241.697103] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6245.266457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6245.431931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6246.938965] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6249.106157] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6250.524092] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6253.213692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6253.314775] LustreError: 173906:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6253.320598] Lustre: 173906:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.203.108@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6253.384642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.663765] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6258.012511] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6260.705453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6260.706285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.715978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.719036] Lustre: Skipped 2 previous similar messages [ 6264.956336] LustreError: 174554:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6264.958678] LustreError: 174554:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6264.984471] Lustre: server umount lustre-MDT0000 complete [ 6265.825947] LustreError: 172321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.833468] LustreError: 172321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6266.549517] LustreError: 172302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759134 with bad export cookie 17919904064913862949 [ 6266.553220] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6266.555685] LustreError: 172302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6266.626296] LustreError: 174756:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6266.629663] LustreError: 174756:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6266.676272] Lustre: server umount lustre-MDT0001 complete [ 6278.579335] LustreError: 174956:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6278.582272] LustreError: 174956:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6278.631182] Lustre: server umount lustre-OST0000 complete [ 6282.352613] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6286.414290] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6301.968372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6302.062722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6302.075971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6302.117438] Lustre: lustre-MDT0000: new disk, initializing [ 6302.148408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6302.160480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6303.744370] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6308.788221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6308.853156] Lustre: 177287:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6308.886384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6308.889070] Lustre: Skipped 1 previous similar message [ 6308.938819] Lustre: lustre-MDT0001: new disk, initializing [ 6308.993337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6309.013699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6309.020360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6310.715815] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6314.448606] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6315.893370] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6318.772191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6318.893253] Lustre: lustre-OST0000: new disk, initializing [ 6318.895636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6318.942954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6321.543597] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6325.383620] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6326.790188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6326.800446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6326.849657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6327.118028] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6327.232890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6328.957307] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6329.056525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6331.872490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6331.873541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6331.875804] LustreError: Skipped 1 previous similar message [ 6331.880267] Lustre: Skipped 2 previous similar messages [ 6331.882229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6331.885628] Lustre: Skipped 1 previous similar message [ 6335.921158] LustreError: 180218:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6335.924395] LustreError: 180218:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6335.968124] Lustre: server umount lustre-OST0000 complete [ 6340.065169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6340.068651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.075386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.114161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.115049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.788958] Lustre: server umount lustre-MDT0000 complete [ 6345.436403] LustreError: 178063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759213 with bad export cookie 17919904064913863649 [ 6345.439539] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6345.441680] LustreError: 178063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6345.545033] LustreError: 180620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6345.548934] LustreError: 180620:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6345.620257] Lustre: server umount lustre-MDT0001 complete [ 6349.544040] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:46:57 (1769759217) [ 6356.871923] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6358.125409] Key type lgssc unregistered [ 6358.287598] LNet: 181802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6358.291615] LNetError: 181802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6358.301340] LNet: Removed LNI 192.168.203.108@tcp [ 6358.622870] Key type .llcrypt unregistered [ 6358.624506] Key type ._llcrypt unregistered [ 6367.047960] Key type ._llcrypt registered [ 6367.049306] Key type .llcrypt registered [ 6367.114323] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6367.545406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6367.572390] alg: No test for adler32 (adler32-zlib) [ 6368.448591] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 6368.580346] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 6370.175164] Key type lgssc registered [ 6370.583523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6373.760433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6373.784041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.029200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.591437] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6380.085945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6380.242186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6381.763139] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6383.998090] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6385.407123] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6388.698339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6388.808857] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6388.810759] LustreError: 184612:0:(llog.c:1684:llog_backup()) MGC192.168.203.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6388.815190] Lustre: 184612:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.203.108@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6388.873707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.284571] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6393.425224] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6395.364129] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.366353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6395.369771] Lustre: Skipped 1 previous similar message [ 6400.391255] LustreError: 185264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6400.405146] LustreError: 185264:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (0000000020a4fcb3) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6400.480183] LustreError: 182978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6400.485131] LustreError: 182978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6402.466378] LustreError: 185264:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6402.469015] LustreError: 185264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6402.488364] Lustre: server umount lustre-MDT0000 complete [ 6403.938335] LustreError: 182959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759272 with bad export cookie 3929999568074075125 [ 6403.942514] LustreError: 182959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6403.942641] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6404.016244] LustreError: 185465:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6404.018534] LustreError: 185465:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6404.078200] Lustre: server umount lustre-MDT0001 complete [ 6415.794711] LustreError: 185667:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6415.797595] LustreError: 185667:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6415.847718] Lustre: server umount lustre-OST0000 complete [ 6419.251515] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6423.042995] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6436.714727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6436.815098] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6436.827931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6436.863889] Lustre: lustre-MDT0000: new disk, initializing [ 6436.907952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6436.917121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6438.416684] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6443.064372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6443.107289] Lustre: 187988:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6443.131271] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6443.132848] Lustre: Skipped 1 previous similar message [ 6443.134588] LustreError: 188752:0:(llog.c:1684:llog_backup()) MGC192.168.203.108@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6443.137662] LustreError: 188752:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 6443.140664] Lustre: 188752:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.108@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6443.150334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6443.152332] Lustre: Skipped 1 previous similar message [ 6443.192525] Lustre: lustre-MDT0001: new disk, initializing [ 6443.252428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6443.272058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6443.276710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6444.844460] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6448.248651] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6449.601373] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6452.132387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6452.220094] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6452.222128] Lustre: Skipped 1 previous similar message [ 6452.224435] LustreError: 189698:0:(llog.c:1684:llog_backup()) MGC192.168.203.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6452.229249] LustreError: 189698:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 6452.231856] Lustre: 189698:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.108@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6452.236213] Lustre: 189698:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6452.246769] Lustre: lustre-OST0000: new disk, initializing [ 6452.248964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6452.293240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6452.348199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6452.353707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6452.369280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 6454.553406] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6458.115049] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6459.688136] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6459.776825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6461.355527] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6461.442534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6462.946246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6462.946908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.950602] LustreError: Skipped 1 previous similar message [ 6462.951582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6462.960597] Lustre: Skipped 2 previous similar messages [ 6462.962643] Lustre: Skipped 2 previous similar messages [ 6464.239178] LustreError: 190913:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6464.243738] LustreError: 190913:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6464.280979] Lustre: server umount lustre-OST0000 complete [ 6468.065425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6468.070082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6468.072013] Lustre: Skipped 2 previous similar messages [ 6471.834469] Lustre: server umount lustre-MDT0000 complete [ 6473.183938] LustreError: 191115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.190166] LustreError: 191115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6473.263124] LustreError: 187979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759341 with bad export cookie 3929999568074075825 [ 6473.266354] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6473.267208] LustreError: 187979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6473.333580] LustreError: 191316:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6473.337131] LustreError: 191316:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6473.396124] Lustre: server umount lustre-MDT0001 complete [ 6476.576073] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:49:04 (1769759344) [ 6482.455932] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6486.058730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6486.200313] LustreError: 192681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.216654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.618507] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6490.970349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6492.597726] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6493.678619] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6496.194303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6496.224706] LustreError: 194129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.485941] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6501.678442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6501.746331] Lustre: lustre-OST0001: new disk, initializing [ 6501.748872] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6503.952065] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6506.497137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6506.500574] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6506.538504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6509.762385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6513.589926] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6519.292932] Lustre: DEBUG MARKER: ADD OST3 [ 6521.053604] loop: module loaded [ 6521.072521] loop0: detected capacity change from 0 to 204800000 [ 6521.204452] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6521.208894] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6521.214756] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6523.625319] loop0: detected capacity change from 0 to 204800000 [ 6525.982782] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6526.058366] Lustre: lustre-OST0003: new disk, initializing [ 6526.060796] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6526.098705] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6526.101485] Lustre: Skipped 3 previous similar messages [ 6526.326898] Lustre: DEBUG MARKER: STOP OST3 [ 6527.064435] Lustre: Failing over lustre-OST0003 [ 6527.088651] LustreError: 196916:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6527.091793] LustreError: 196916:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6527.111534] Lustre: server umount lustre-OST0003 complete [ 6527.190259] LustreError: 195449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6528.501799] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6532.249444] Lustre: DEBUG MARKER: START OST3 again [ 6534.938360] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6537.013669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6537.017018] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6537.038196] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 6537.147737] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6550.890461] Lustre: Failing over lustre-OST0003 [ 6550.930538] Lustre: server umount lustre-OST0003 complete [ 6552.544890] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6552.546792] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.547901] LustreError: Skipped 1 previous similar message [ 6552.550649] LustreError: 194129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6552.551959] Lustre: Skipped 2 previous similar messages [ 6552.561105] LustreError: 194129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6565.860282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.861528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6565.864649] Lustre: Skipped 2 previous similar messages [ 6565.866201] Lustre: Skipped 1 previous similar message [ 6569.742312] LustreError: 199225:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 6569.744992] LustreError: 199225:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6569.787367] Lustre: server umount lustre-MDT0000 complete [ 6570.977548] LustreError: 192680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6570.982379] LustreError: 192680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6571.115754] LustreError: 192660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759439 with bad export cookie 3929999568074076665 [ 6571.119839] LustreError: 192660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6571.121021] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6576.097805] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6576.098602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6576.102824] Lustre: Skipped 2 previous similar messages [ 6576.104804] Lustre: Skipped 2 previous similar messages [ 6577.332263] Lustre: server umount lustre-MDT0001 complete [ 6578.759904] Lustre: server umount lustre-OST0000 complete [ 6580.138492] Lustre: server umount lustre-OST0001 complete [ 6582.431353] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6585.331229] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6598.209892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.310906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6598.324536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6598.364531] Lustre: lustre-MDT0000: new disk, initializing [ 6598.387314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.389392] Lustre: Skipped 1 previous similar message [ 6598.394871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6599.816303] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6604.409282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6604.460235] Lustre: 201999:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6604.577097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6606.048302] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6609.439908] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6610.765162] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6613.167838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6614.521331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6614.524563] Lustre: Skipped 1 previous similar message [ 6614.526795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6614.554397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6615.251593] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6618.608592] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6620.101563] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6620.186281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6621.616268] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6621.692239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6624.738604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6624.738629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.739333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6624.739338] Lustre: Skipped 1 previous similar message [ 6624.742922] LustreError: Skipped 1 previous similar message [ 6624.752183] Lustre: Skipped 1 previous similar message [ 6628.433159] Lustre: server umount lustre-OST0000 complete [ 6644.191162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6644.234563] LustreError: 205130:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6644.237228] LustreError: 205130:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 6644.267147] Lustre: server umount lustre-MDT0000 complete [ 6645.216208] LustreError: 202010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.222891] LustreError: 202010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6645.773265] LustreError: 202777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759513 with bad export cookie 3929999568074078926 [ 6645.775918] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6645.782229] LustreError: 202777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6645.910460] Lustre: server umount lustre-MDT0001 complete [ 6647.309399] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:51:55 (1769759515) [ 6652.870495] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6656.244254] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6668.434504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6668.525931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6668.540692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6668.542970] Lustre: Skipped 3 previous similar messages [ 6668.583746] Lustre: lustre-MDT0000: new disk, initializing [ 6668.586517] Lustre: Skipped 2 previous similar messages [ 6668.627325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6669.986303] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6674.527400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6674.573433] Lustre: 208158:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6674.689095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6676.064574] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6679.160490] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6680.324443] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6682.668218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6684.259394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6684.740833] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6687.958546] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6689.451111] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6689.531317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6690.890147] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6690.972755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6694.370109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6694.370940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.374718] LustreError: Skipped 1 previous similar message [ 6694.380426] Lustre: Skipped 2 previous similar messages [ 6694.381139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6694.387929] Lustre: Skipped 8 previous similar messages [ 6697.747082] Lustre: server umount lustre-OST0000 complete [ 6705.253256] Lustre: server umount lustre-MDT0000 complete [ 6706.739258] LustreError: 208149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759574 with bad export cookie 3929999568074079808 [ 6706.742517] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6706.744131] LustreError: 208149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6706.871349] Lustre: server umount lustre-MDT0001 complete [ 6709.936889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.122884] LustreError: 211967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.566214] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6714.474438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6715.921885] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6717.740326] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6718.887773] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6721.271651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6723.364904] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6725.245853] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6750.176305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6750.179643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.182474] Lustre: Skipped 4 previous similar messages [ 6753.235132] Lustre: server umount lustre-OST0000 complete [ 6760.791690] Lustre: server umount lustre-MDT0000 complete [ 6762.056775] LustreError: 211992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759630 with bad export cookie 3929999568074080690 [ 6762.057280] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.064274] LustreError: 211992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6762.177152] Lustre: server umount lustre-MDT0001 complete [ 6858.955377] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 6859.845164] Key type lgssc unregistered [ 6859.965281] LNet: 215295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6859.967460] LNetError: 215295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6859.973271] LNet: Removed LNI 192.168.203.108@tcp [ 6860.253145] Key type .llcrypt unregistered [ 6860.254338] Key type ._llcrypt unregistered [ 6868.031297] Key type ._llcrypt registered [ 6868.032736] Key type .llcrypt registered [ 6868.069623] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 6874.370647] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6874.682698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6874.707080] alg: No test for adler32 (adler32-zlib) [ 6875.555573] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 6875.640031] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 6877.215172] Key type lgssc registered [ 6877.582290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6889.874169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6889.880820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.988574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6891.003293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6891.042519] Lustre: lustre-MDT0000: new disk, initializing [ 6891.084650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6891.095263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6892.590254] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6896.886136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6896.932354] Lustre: 218582:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6896.959261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6896.961664] Lustre: Skipped 1 previous similar message [ 6897.000874] Lustre: lustre-MDT0001: new disk, initializing [ 6897.041782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6897.055339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6897.060146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6898.562988] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6901.688733] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6903.082426] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6905.495994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6905.593473] Lustre: lustre-OST0000: new disk, initializing [ 6905.596172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6905.627112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.603958] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6910.792493] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6912.212841] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6913.528859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6913.533889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6913.560497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6914.316380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6915.649988] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6915.729062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6918.625595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6918.626381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.628434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6918.631761] Lustre: Skipped 1 previous similar message [ 6918.635770] Lustre: Skipped 1 previous similar message [ 6922.479101] LustreError: 221528:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6922.519993] Lustre: server umount lustre-OST0000 complete [ 6927.840026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6927.842853] LustreError: Skipped 1 previous similar message [ 6927.844243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.849956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6928.863996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6928.864546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.984303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6933.986605] Lustre: Skipped 2 previous similar messages [ 6938.079170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6938.080343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6938.083458] Lustre: Skipped 1 previous similar message [ 6938.112746] LustreError: 221729:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6938.114493] LustreError: 221729:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6938.148686] Lustre: server umount lustre-MDT0000 complete [ 6939.329268] LustreError: 218575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759807 with bad export cookie 2121776529337779109 [ 6939.332831] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6939.333371] LustreError: 218575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6939.393432] LustreError: 221931:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6939.395398] LustreError: 221931:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6939.447933] Lustre: server umount lustre-MDT0001 complete [ 6945.546647] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 6949.090425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6949.278388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6950.603795] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6953.442792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6953.567901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6954.844302] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6955.772257] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6957.981245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6959.852041] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6962.540716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6962.600701] Lustre: lustre-OST0001: new disk, initializing [ 6962.602498] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6962.636903] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6962.639502] Lustre: Skipped 1 previous similar message [ 6963.453059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6963.456153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6963.484552] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6964.481885] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 6969.963516] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6972.611306] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6981.140328] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:57:29 (1769759849) [ 6984.160146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6984.164054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.169633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6987.901585] LustreError: 226473:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6987.903529] LustreError: 226473:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6987.932837] Lustre: server umount lustre-MDT0000 complete [ 6989.167125] LustreError: 224204:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759857 with bad export cookie 2121776529337779977 [ 6989.170960] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6989.171733] LustreError: 224204:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6989.280495] Lustre: server umount lustre-MDT0001 complete [ 7000.690683] LustreError: 226875:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7000.692964] LustreError: 226875:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7000.709568] Lustre: server umount lustre-OST0000 complete [ 7012.209198] LustreError: 227076:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7012.211025] LustreError: 227076:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7012.257046] Lustre: server umount lustre-OST0001 complete [ 7014.226764] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 7017.314694] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7028.988068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.066881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7029.078206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7029.116206] Lustre: lustre-MDT0000: new disk, initializing [ 7029.140813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.148085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7030.407729] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7034.509281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7034.547084] Lustre: 229246:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7034.639603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7035.947871] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7038.981571] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7040.153228] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7042.394695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7042.495128] Lustre: lustre-OST0000: new disk, initializing [ 7042.496379] Lustre: Skipped 1 previous similar message [ 7042.497830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7042.502584] Lustre: Skipped 2 previous similar messages [ 7044.346289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7044.348978] Lustre: Skipped 1 previous similar message [ 7044.349250] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7044.351129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7044.380898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7047.245905] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7048.403119] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7048.468378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7049.649130] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7049.711876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7054.816849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7054.817899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7054.819633] LustreError: Skipped 1 previous similar message [ 7054.822624] Lustre: Skipped 3 previous similar messages [ 7054.823367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7054.827130] Lustre: Skipped 5 previous similar messages [ 7056.364302] LustreError: 232179:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7056.366800] LustreError: 232179:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7056.398921] Lustre: server umount lustre-OST0000 complete [ 7063.633993] Lustre: server umount lustre-MDT0000 complete [ 7064.883383] LustreError: 229239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759933 with bad export cookie 2121776529337782077 [ 7064.886248] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7064.888804] LustreError: 229239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7065.012459] Lustre: server umount lustre-MDT0001 complete [ 7067.841557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.983652] LustreError: 233058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.990063] LustreError: 233058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7068.009943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.012037] Lustre: Skipped 2 previous similar messages [ 7069.352122] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7072.136338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7073.600341] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7075.488657] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7076.684595] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7078.912417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7080.844508] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7082.566682] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7225.824362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7225.826669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.829600] Lustre: Skipped 3 previous similar messages [ 7225.831447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7225.834362] Lustre: Skipped 2 previous similar messages [ 7230.638132] LustreError: 235351:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7230.641447] LustreError: 235351:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7230.678064] Lustre: server umount lustre-OST0000 complete [ 7246.303127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7246.359271] Lustre: server umount lustre-MDT0000 complete [ 7247.327892] LustreError: 234376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.332700] LustreError: 234376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7247.562065] LustreError: 234672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760115 with bad export cookie 2121776529337782959 [ 7247.565493] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7247.565924] LustreError: 234672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7247.675933] Lustre: server umount lustre-MDT0001 complete [ 7346.483352] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 7347.474094] Key type lgssc unregistered [ 7347.607308] LNet: 236321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7347.610570] LNetError: 236321:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7347.618286] LNet: Removed LNI 192.168.203.108@tcp [ 7347.895100] Key type .llcrypt unregistered [ 7347.896290] Key type ._llcrypt unregistered [ 7355.601356] Key type ._llcrypt registered [ 7355.603075] Key type .llcrypt registered [ 7355.642670] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 7361.839736] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7362.148393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7362.183046] alg: No test for adler32 (adler32-zlib) [ 7363.033232] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 7363.111618] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 7364.687152] Key type lgssc registered [ 7365.039640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7379.239873] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7382.794133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7382.801042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.894855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7383.907050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7383.948730] Lustre: lustre-MDT0000: new disk, initializing [ 7383.975509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.983799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7385.352760] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7389.637782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7389.680304] Lustre: 240306:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7389.705437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7389.708138] Lustre: Skipped 1 previous similar message [ 7389.743682] Lustre: lustre-MDT0001: new disk, initializing [ 7389.776415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7389.788112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7389.793697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7391.132969] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7393.388877] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7395.659653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7395.766199] Lustre: lustre-OST0000: new disk, initializing [ 7395.768974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7395.806572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.797460] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7401.470357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7401.475122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7401.525990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7402.184194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7402.250455] Lustre: lustre-OST0001: new disk, initializing [ 7402.252761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7402.288610] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7404.203689] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7408.125938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7408.131198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7408.157751] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7409.580962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7413.179237] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7423.457969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.459540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7423.461379] Lustre: Skipped 2 previous similar messages [ 7423.464368] Lustre: Skipped 3 previous similar messages [ 7427.206380] LustreError: 243824:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7427.238261] Lustre: server umount lustre-MDT0000 complete [ 7428.435204] LustreError: 240297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760296 with bad export cookie 4081433943988141916 [ 7428.438489] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.439720] LustreError: 240297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7428.507956] LustreError: 244026:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7428.509865] LustreError: 244026:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7428.559580] Lustre: server umount lustre-MDT0001 complete [ 7440.179273] LustreError: 244226:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7440.181726] LustreError: 244226:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7440.201184] Lustre: server umount lustre-OST0000 complete [ 7451.698288] LustreError: 244428:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7451.700969] LustreError: 244428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7451.746941] Lustre: server umount lustre-OST0001 complete [ 7456.187674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7459.153779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7462.374700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7465.398968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7471.317116] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7475.250477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.260229] Lustre: lustre-MDT0000: reset Object Index mappings [ 7475.522412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.274139] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7480.805423] LustreError: 247730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.821631] LustreError: 247730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7484.902104] LustreError: 247730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7485.138795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7485.148084] Lustre: lustre-MDT0001: reset Object Index mappings [ 7485.445429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7488.624786] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7490.512279] Lustre: 248837:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7494.736944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7496.047337] LustreError: 249193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.398520] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7500.133121] LustreError: 249295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7500.155886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7500.159281] LustreError: 249295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7505.385474] LustreError: 249295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.391460] LustreError: 249295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7506.191849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7506.419690] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7506.422875] Lustre: Skipped 1 previous similar message [ 7508.321083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7511.090519] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7516.531565] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7519.142644] Lustre: 250683:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7531.488050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7531.495270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7531.514076] Lustre: Skipped 1 previous similar message [ 7531.517918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7533.580265] LustreError: 250786:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7533.583107] LustreError: 250786:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7533.673322] Lustre: server umount lustre-MDT0000 complete [ 7535.530884] LustreError: 247711:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760403 with bad export cookie 4081433943988148692 [ 7535.531407] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.535442] LustreError: 247711:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7535.727277] Lustre: server umount lustre-MDT0001 complete [ 7548.222142] LustreError: 251190:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7548.226356] LustreError: 251190:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7548.258421] Lustre: server umount lustre-OST0000 complete [ 7560.377850] Lustre: server umount lustre-OST0001 complete [ 7569.430089] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:07:17 (1769760437) [ 7577.139849] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 7581.745274] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7600.301239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7600.475256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7600.498236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7600.550339] Lustre: lustre-MDT0000: new disk, initializing [ 7600.605128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7600.615870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7602.686139] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7608.157392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7608.208560] Lustre: 254900:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7608.373548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7610.108161] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7614.257942] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7615.892585] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7618.950733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7619.072510] Lustre: lustre-OST0000: new disk, initializing [ 7619.073958] Lustre: Skipped 1 previous similar message [ 7619.075897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7619.078823] Lustre: Skipped 2 previous similar messages [ 7620.163497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7620.167934] Lustre: Skipped 1 previous similar message [ 7620.170353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7620.224241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7621.890027] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7625.554341] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7627.182169] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7627.269793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7628.747668] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7628.837756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7630.307638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7630.311296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.316399] Lustre: Skipped 3 previous similar messages [ 7630.319181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7630.322640] Lustre: Skipped 4 previous similar messages [ 7635.632484] LustreError: 257825:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7635.635124] LustreError: 257825:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7635.681433] Lustre: server umount lustre-OST0000 complete [ 7639.007902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7639.011444] LustreError: Skipped 1 previous similar message [ 7639.013347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.018759] Lustre: Skipped 1 previous similar message [ 7639.020799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7639.022660] Lustre: Skipped 1 previous similar message [ 7643.581151] Lustre: server umount lustre-MDT0000 complete [ 7645.239412] LustreError: 256647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760513 with bad export cookie 4081433943988150785 [ 7645.242139] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7645.243151] LustreError: 256647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7645.386830] Lustre: server umount lustre-MDT0001 complete [ 7652.848073] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7657.997891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.208690] LustreError: 259403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.234227] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7664.729719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7664.983852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7664.989925] Lustre: Skipped 3 previous similar messages [ 7667.490618] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7668.992517] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7672.651446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7676.274388] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7681.293036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7681.402674] Lustre: lustre-OST0001: new disk, initializing [ 7681.405694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7682.749579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7682.756412] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7682.792567] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7684.709731] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7691.229491] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7693.171926] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7700.645927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7700.670374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7704.554071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7704.559051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7704.563641] Lustre: Skipped 4 previous similar messages [ 7704.567101] Lustre: Skipped 4 previous similar messages [ 7709.644123] LustreError: 262898:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7709.647843] LustreError: 262898:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7709.667389] LustreError: 259399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.678876] LustreError: 259399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7709.702086] Lustre: server umount lustre-MDT0000 complete [ 7711.491913] LustreError: 259382:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760579 with bad export cookie 4081433943988151667 [ 7711.495387] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.496485] LustreError: 259382:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7711.682729] Lustre: server umount lustre-MDT0001 complete [ 7723.009253] Lustre: server umount lustre-OST0000 complete [ 7734.689750] Lustre: server umount lustre-OST0001 complete [ 7740.605352] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7744.471852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.628133] LustreError: 264675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7746.063282] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7749.161652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7750.608688] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7751.608948] Lustre: 265779:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7753.867546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7755.063551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7755.829152] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7758.866348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7759.981312] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7760.905440] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7764.046069] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7765.307142] Lustre: 267628:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7771.926434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7771.937994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7774.864034] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:10:43 (1769760643) [ 7790.559879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7790.562500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.566649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7790.569455] Lustre: Skipped 1 previous similar message [ 7792.645297] LustreError: 268334:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7792.649017] LustreError: 268334:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7792.681208] Lustre: server umount lustre-MDT0000 complete [ 7794.012534] LustreError: 266949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760662 with bad export cookie 4081433943988157981 [ 7794.016345] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.016937] LustreError: 266949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7800.181642] Lustre: server umount lustre-MDT0001 complete [ 7807.623796] Lustre: server umount lustre-OST0000 complete [ 7823.327147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7823.386912] Lustre: server umount lustre-OST0001 complete [ 7825.496676] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 7828.940470] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7846.341185] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7850.412152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.525378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7850.545536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7850.585323] Lustre: lustre-MDT0000: new disk, initializing [ 7850.634228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.636989] Lustre: Skipped 6 previous similar messages [ 7850.652186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7852.205661] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7856.663846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7856.704462] Lustre: 271807:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7856.816481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7858.180404] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7862.690572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7863.552300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7864.825929] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7869.155535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7870.471609] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7871.591857] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7876.324402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7886.721451] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 03:12:34 (1769760754) [ 7887.839953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7887.843178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7887.846770] Lustre: Skipped 5 previous similar messages [ 7887.849125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7887.851319] Lustre: Skipped 8 previous similar messages [ 7893.593981] Lustre: server umount lustre-MDT0000 complete [ 7894.909303] LustreError: 271798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760763 with bad export cookie 4081433943988165856 [ 7894.913054] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7894.913579] LustreError: 271798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7895.035029] Lustre: server umount lustre-MDT0001 complete [ 7906.695761] Lustre: server umount lustre-OST0000 complete [ 7918.170883] Lustre: server umount lustre-OST0001 complete [ 7924.191270] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 7927.359686] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 7939.701792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7939.796290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7939.800214] Lustre: Skipped 2 previous similar messages [ 7939.810933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7939.813675] Lustre: Skipped 4 previous similar messages [ 7939.856545] Lustre: lustre-MDT0000: new disk, initializing [ 7939.858170] Lustre: Skipped 3 previous similar messages [ 7939.894641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7939.897780] Lustre: Skipped 3 previous similar messages [ 7941.245965] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7945.529792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7945.570102] Lustre: 278842:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7945.679440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7945.683114] Lustre: Skipped 2 previous similar messages [ 7947.016623] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7950.138845] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7951.282327] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7953.490967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7955.172536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7955.524394] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 7958.548839] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7959.298482] Lustre: Mounted lustre-client [ 7959.634126] LustreError: 281336:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8af20bf56800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 7959.639096] LustreError: 281336:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7959.641483] LustreError: 281336:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 7959.663143] Lustre: Unmounted lustre-client [ 7964.642536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.646800] Lustre: Skipped 3 previous similar messages [ 7964.648838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7964.650753] Lustre: Skipped 5 previous similar messages [ 7968.788366] Lustre: server umount lustre-OST0000 complete [ 7971.295906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7976.211797] Lustre: server umount lustre-MDT0000 complete [ 7977.422153] LustreError: 278832:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760845 with bad export cookie 4081433943988167382 [ 7977.425255] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7977.426428] LustreError: 278832:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7977.540449] Lustre: server umount lustre-MDT0001 complete [ 7981.044711] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 7982.053667] Key type lgssc unregistered [ 8018.463157] LNet: 1 local NIs in recovery (showing 1): 192.168.203.108@tcp [ 8019.039334] LNet: 282614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8019.041921] LNetError: 282614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8090.725471] LNet: Removed LNI 192.168.203.108@tcp [ 8091.043133] Key type .llcrypt unregistered [ 8091.045249] Key type ._llcrypt unregistered [ 8096.911321] Key type ._llcrypt registered [ 8096.912366] Key type .llcrypt registered [ 8096.951296] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 8103.463093] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 8103.772576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8103.779259] alg: No test for adler32 (adler32-zlib) [ 8104.627070] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8104.706246] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 8106.295150] Key type lgssc registered [ 8106.665973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8116.585678] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:16:24 (1769760984) [ 8118.897553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8118.905147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8119.996163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8120.008301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8120.049527] Lustre: lustre-MDT0000: new disk, initializing [ 8120.093288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.102326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8121.471805] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8125.747103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8125.791562] Lustre: 285769:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8125.816948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8125.820729] Lustre: Skipped 1 previous similar message [ 8125.854468] Lustre: lustre-MDT0001: new disk, initializing [ 8125.888307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8125.900840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8125.904597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8127.243491] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8130.543439] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8131.883582] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8134.446105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8134.551203] Lustre: lustre-OST0000: new disk, initializing [ 8134.553544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8134.585974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8136.607592] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8139.720630] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8140.278758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8140.282086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8140.308775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8150.496563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8150.497277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.499937] LustreError: Skipped 1 previous similar message [ 8150.507318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8150.510450] Lustre: Skipped 1 previous similar message [ 8155.117238] LustreError: 288207:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8155.160284] Lustre: server umount lustre-OST0000 complete [ 8156.639666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8156.641540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8156.644360] Lustre: Skipped 1 previous similar message [ 8156.645984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8160.736742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8160.737132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8162.495655] LustreError: 288409:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8162.498290] LustreError: 288409:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8162.525453] Lustre: server umount lustre-MDT0000 complete [ 8163.711103] LustreError: 285761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761031 with bad export cookie 16930415156778885381 [ 8163.714292] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8163.714610] LustreError: 285761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8163.772829] LustreError: 288612:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8163.775536] LustreError: 288612:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8163.825594] Lustre: server umount lustre-MDT0001 complete [ 8167.351632] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8168.489398] Key type lgssc unregistered [ 8168.618308] LNet: 289142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8168.620457] LNetError: 289142:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8168.628265] LNet: Removed LNI 192.168.203.108@tcp [ 8168.902170] Key type .llcrypt unregistered [ 8168.903044] Key type ._llcrypt unregistered [ 8177.581426] Key type ._llcrypt registered [ 8177.582249] Key type .llcrypt registered [ 8177.613468] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 8178.032167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8178.048249] alg: No test for adler32 (adler32-zlib) [ 8178.899631] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8178.977038] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 8180.551137] Key type lgssc registered [ 8180.920324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8183.587043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8183.594419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.754060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8186.105196] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8188.889452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8189.006238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8190.304450] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8192.039682] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8193.201423] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8195.531118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8195.645258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8197.603929] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8199.449252] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8217.590148] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8218.117918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8219.615833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8219.618653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.623667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.641910] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.642487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.647374] Lustre: Skipped 1 previous similar message [ 8220.649576] Lustre: Skipped 2 previous similar messages [ 8221.563145] LustreError: 293019:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8221.591303] Lustre: server umount lustre-MDT0000 complete [ 8222.802135] LustreError: 290497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761090 with bad export cookie 1959640528613103477 [ 8222.805559] LustreError: 290497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8222.806031] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8224.892715] LustreError: 293222:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8224.895022] LustreError: 293222:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8224.937932] Lustre: server umount lustre-MDT0001 complete [ 8242.475894] LustreError: 293424:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8242.477761] LustreError: 293424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8242.524023] Lustre: server umount lustre-OST0000 complete [ 8245.267496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.447979] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8247.346943] Lustre: 294267:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.108@tcp [ 8247.984785] LustreError: 294364:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8247.987424] LustreError: 294364:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8248.010099] Lustre: server umount MGS complete [ 8250.754015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8250.894717] LustreError: 294839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.900673] LustreError: 294839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8250.914558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8252.095165] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8254.721525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8256.051393] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8258.696622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8258.801111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.803128] Lustre: Skipped 1 previous similar message [ 8260.611202] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8262.290814] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8262.785407] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8263.852364] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8266.455779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8269.280831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8269.282613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8269.284790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8269.288361] Lustre: Skipped 1 previous similar message [ 8269.292950] Lustre: Skipped 1 previous similar message [ 8273.643089] LustreError: 297078:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8273.645745] LustreError: 297078:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8273.680909] Lustre: server umount lustre-OST0000 complete [ 8275.423807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8275.425845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8275.429466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8279.520023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8279.520668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.247124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8289.277848] LustreError: 297281:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8289.279362] LustreError: 297281:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8289.298801] Lustre: server umount lustre-MDT0000 complete [ 8289.760176] LustreError: 294857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8289.766979] LustreError: 294857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8290.524357] LustreError: 294819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761158 with bad export cookie 1959640528613104303 [ 8290.528273] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8290.529783] LustreError: 294819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8290.637279] Lustre: server umount lustre-MDT0001 complete [ 8294.094322] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8295.133755] Key type lgssc unregistered [ 8295.256409] LNet: 298016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8295.259606] LNetError: 298016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8295.268279] LNet: Removed LNI 192.168.203.108@tcp [ 8295.555539] Key type .llcrypt unregistered [ 8295.556506] Key type ._llcrypt unregistered [ 8301.254948] Key type ._llcrypt registered [ 8301.255749] Key type .llcrypt registered [ 8301.290414] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 8307.651217] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 8308.065696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8308.097994] alg: No test for adler32 (adler32-zlib) [ 8308.943788] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8309.038922] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 8310.615154] Key type lgssc registered [ 8310.947396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8322.340647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8322.345662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.432728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8323.444424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8323.478040] Lustre: lustre-MDT0000: new disk, initializing [ 8323.503588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8323.510595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8324.871497] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8329.155356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8329.200320] Lustre: 301108:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8329.222425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8329.224622] Lustre: Skipped 1 previous similar message [ 8329.263247] Lustre: lustre-MDT0001: new disk, initializing [ 8329.305053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8329.324214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8329.330029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8330.660742] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8333.824416] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8335.011197] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8337.251197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8337.354130] Lustre: lustre-OST0000: new disk, initializing [ 8337.355739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8337.361525] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8337.402289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.291812] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 8342.386299] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8342.506989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8342.510635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8342.524691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8343.853237] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8343.928932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8345.358916] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8345.438939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8347.616442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8347.616889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8347.621420] LustreError: Skipped 1 previous similar message [ 8347.622113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8347.628414] Lustre: Skipped 1 previous similar message [ 8352.171092] LustreError: 304037:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 8352.215694] Lustre: server umount lustre-OST0000 complete [ 8354.783905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8354.787215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.792996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.795180] Lustre: Skipped 1 previous similar message [ 8357.856025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8357.856615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8359.615797] LustreError: 304243:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8359.617456] LustreError: 304243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8359.643576] Lustre: server umount lustre-MDT0000 complete [ 8360.860120] LustreError: 301101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761229 with bad export cookie 4096933238096648219 [ 8360.863165] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8360.863582] LustreError: 301101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8360.927242] LustreError: 304445:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8360.929418] LustreError: 304445:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8360.979467] Lustre: server umount lustre-MDT0001 complete [ 8363.594860] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 03:20:31 (1769761231) [ 8380.639430] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 03:20:48 (1769761248) [ 8394.506275] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8395.470529] Key type lgssc unregistered [ 8395.590380] LNet: 307649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8395.593046] LNetError: 307649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8395.599283] LNet: Removed LNI 192.168.203.108@tcp [ 8395.883218] Key type .llcrypt unregistered [ 8395.884426] Key type ._llcrypt unregistered [ 8422.217529] Key type ._llcrypt registered [ 8422.218634] Key type .llcrypt registered [ 8422.255366] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8422.357404] Key type .llcrypt unregistered [ 8422.358636] Key type ._llcrypt unregistered [ 8433.098878] Key type ._llcrypt registered [ 8433.100367] Key type .llcrypt registered [ 8433.134212] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8433.225326] Key type .llcrypt unregistered [ 8433.226284] Key type ._llcrypt unregistered [ 8443.622401] Key type ._llcrypt registered [ 8443.623332] Key type .llcrypt registered [ 8443.664568] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8443.753442] Key type .llcrypt unregistered [ 8443.754821] Key type ._llcrypt unregistered [ 8471.695215] Key type ._llcrypt registered [ 8471.696190] Key type .llcrypt registered [ 8471.731499] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8471.832327] Key type .llcrypt unregistered [ 8471.833364] Key type ._llcrypt unregistered [ 8482.552497] Key type ._llcrypt registered [ 8482.553535] Key type .llcrypt registered [ 8482.587597] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8482.676254] Key type .llcrypt unregistered [ 8482.677203] Key type ._llcrypt unregistered [ 8493.195294] Key type ._llcrypt registered [ 8493.197180] Key type .llcrypt registered [ 8493.231313] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8493.321268] Key type .llcrypt unregistered [ 8493.322367] Key type ._llcrypt unregistered [ 8503.597728] Key type ._llcrypt registered [ 8503.599215] Key type .llcrypt registered [ 8503.633703] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8503.722169] Key type .llcrypt unregistered [ 8503.723129] Key type ._llcrypt unregistered [ 8514.512391] Key type ._llcrypt registered [ 8514.513217] Key type .llcrypt registered [ 8514.547383] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8514.641318] Key type .llcrypt unregistered [ 8514.642356] Key type ._llcrypt unregistered [ 8525.666554] Key type ._llcrypt registered [ 8525.668071] Key type .llcrypt registered [ 8525.704986] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8525.793561] Key type .llcrypt unregistered [ 8525.794770] Key type ._llcrypt unregistered [ 8546.779622] Key type ._llcrypt registered [ 8546.780822] Key type .llcrypt registered [ 8546.821630] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8546.925285] Key type .llcrypt unregistered [ 8546.927027] Key type ._llcrypt unregistered [ 8558.101983] Key type ._llcrypt registered [ 8558.103679] Key type .llcrypt registered [ 8558.143018] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8558.239212] Key type .llcrypt unregistered [ 8558.240079] Key type ._llcrypt unregistered [ 8568.680137] Key type ._llcrypt registered [ 8568.681680] Key type .llcrypt registered [ 8568.714957] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8568.802376] Key type .llcrypt unregistered [ 8568.803902] Key type ._llcrypt unregistered [ 8586.631533] Key type ._llcrypt registered [ 8586.632876] Key type .llcrypt registered [ 8586.669545] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 8592.673445] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 8592.997357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8593.042253] alg: No test for adler32 (adler32-zlib) [ 8593.884742] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 8593.965519] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 8595.543138] Key type lgssc registered [ 8595.894676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8605.669630] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8344 sec ========== 03:24:33 (1769761473) [ 8606.249795] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:24:34 (1769761474) === [ 8607.389756] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:24:35 (1769761475) === [ 8619.833571] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 8620.800092] Key type lgssc unregistered [ 8620.914347] LNet: 326427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8620.916493] LNetError: 326427:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8620.924319] LNet: Removed LNI 192.168.203.108@tcp [ 8621.189438] Key type .llcrypt unregistered [ 8621.190808] Key type ._llcrypt unregistered