[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481734049 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2608MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003153] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008565] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011148] LSM: Security Framework initializing [ 0.013051] Yama: becoming mindful. [ 0.014065] SELinux: Initializing. [ 0.015086] *** VALIDATE selinux *** [ 0.023807] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028787] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031131] *** VALIDATE tmpfs *** [ 0.033419] *** VALIDATE proc *** [ 0.035098] *** VALIDATE cgroup *** [ 0.036000] *** VALIDATE cgroup2 *** [ 0.037206] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040037] Spectre V2 : User space: Vulnerable [ 0.041014] Speculative Store Bypass: Vulnerable [ 0.044629] debug: unmapping init [mem 0xffffffffa1859000-0xffffffffa1860fff] [ 0.045862] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047353] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048031] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052019] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055234] rcu: Hierarchical SRCU implementation. [ 0.057357] smp: Bringing up secondary CPUs ... [ 0.058659] x86: Booting SMP configuration: [ 0.059032] .... node #0, CPUs: #1 #2 #3 [ 0.063436] smp: Brought up 1 node, 4 CPUs [ 0.064996] smpboot: Max logical packages: 1 [ 0.065016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150037] node 0 deferred pages initialised in 82ms [ 0.153166] devtmpfs: initialized [ 0.155212] x86/mm: Memory block size: 128MB [ 0.157845] gcov: version magic: 0x41383552 [ 0.162028] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164313] pinctrl core: initialized pinctrl subsystem [ 0.165186] [ 0.165809] ************************************************************* [ 0.166017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167011] ** ** [ 0.168018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169018] ** ** [ 0.170015] ** This means that this kernel is built to expose internal ** [ 0.171032] ** IOMMU data structures, which may compromise security on ** [ 0.172016] ** your system. ** [ 0.173017] ** ** [ 0.174015] ** If you see this message and you are not debugging the ** [ 0.175015] ** kernel, report this immediately to your vendor! ** [ 0.176013] ** ** [ 0.177017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178014] ************************************************************* [ 0.179711] NET: Registered protocol family 16 [ 0.180470] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.184032] cpuidle: using governor menu [ 0.185640] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188517] PCI: Using configuration type 1 for base access [ 0.191140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206111] cryptd: max_cpu_qlen set to 1000 [ 0.209241] ACPI: Added _OSI(Module Device) [ 0.210013] ACPI: Added _OSI(Processor Device) [ 0.211013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215654] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218499] ACPI: Interpreter enabled [ 0.219057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220015] ACPI: Using IOAPIC for interrupt routing [ 0.221141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231378] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236395] acpiphp: Slot [2] registered [ 0.237150] acpiphp: Slot [5] registered [ 0.238154] acpiphp: Slot [6] registered [ 0.239127] acpiphp: Slot [7] registered [ 0.240146] acpiphp: Slot [8] registered [ 0.241118] acpiphp: Slot [9] registered [ 0.242147] acpiphp: Slot [10] registered [ 0.244164] acpiphp: Slot [3] registered [ 0.245115] acpiphp: Slot [4] registered [ 0.247130] acpiphp: Slot [11] registered [ 0.248133] acpiphp: Slot [12] registered [ 0.249139] acpiphp: Slot [13] registered [ 0.251152] acpiphp: Slot [14] registered [ 0.253111] acpiphp: Slot [15] registered [ 0.254128] acpiphp: Slot [16] registered [ 0.256117] acpiphp: Slot [17] registered [ 0.258128] acpiphp: Slot [18] registered [ 0.259108] acpiphp: Slot [19] registered [ 0.261140] acpiphp: Slot [20] registered [ 0.262110] acpiphp: Slot [21] registered [ 0.264133] acpiphp: Slot [22] registered [ 0.266129] acpiphp: Slot [23] registered [ 0.268126] acpiphp: Slot [24] registered [ 0.269000] acpiphp: Slot [25] registered [ 0.271130] acpiphp: Slot [26] registered [ 0.273179] acpiphp: Slot [27] registered [ 0.275116] acpiphp: Slot [28] registered [ 0.276109] acpiphp: Slot [29] registered [ 0.278140] acpiphp: Slot [30] registered [ 0.280128] acpiphp: Slot [31] registered [ 0.281098] PCI host bridge to bus 0000:00 [ 0.283017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.285046] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303222] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306440] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.322854] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.331022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.333688] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.337084] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.340052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.343801] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363746] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367980] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375457] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.399015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.432011] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.443017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.452017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.471020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484080] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.502016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.520033] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.532182] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.539019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.548019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.574019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.586775] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.593018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.598016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.610017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.620643] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.626017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.632018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.649016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.660248] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.662375] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.665348] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.667380] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.670228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.675102] iommu: Default domain type: Passthrough [ 0.677502] SCSI subsystem initialized [ 0.679208] ACPI: bus type USB registered [ 0.680115] usbcore: registered new interface driver usbfs [ 0.683111] usbcore: registered new interface driver hub [ 0.684106] usbcore: registered new device driver usb [ 0.686215] pps_core: LinuxPPS API ver. 1 registered [ 0.688013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.691076] PTP clock support registered [ 0.694048] EDAC MC: Ver: 3.0.0 [ 0.696143] PCI: Using ACPI for IRQ routing [ 0.698936] NetLabel: Initializing [ 0.700012] NetLabel: domain hash size = 128 [ 0.701009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.703086] NetLabel: unlabeled traffic allowed by default [ 0.705150] vgaarb: loaded [ 0.707209] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.709019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.716762] clocksource: Switched to clocksource kvm-clock [ 0.826291] VFS: Disk quotas dquot_6.6.0 [ 0.827970] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.830445] *** VALIDATE ramfs *** [ 0.831511] *** VALIDATE hugetlbfs *** [ 0.833071] pnp: PnP ACPI init [ 0.835310] pnp: PnP ACPI: found 6 devices [ 0.852375] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.855131] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.856995] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859093] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861511] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.863896] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.866490] NET: Registered protocol family 2 [ 0.868943] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.874038] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.877969] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.884273] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.887875] TCP: Hash tables configured (established 65536 bind 65536) [ 0.890894] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.895454] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898479] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.901406] NET: Registered protocol family 1 [ 0.904118] RPC: Registered named UNIX socket transport module. [ 0.906200] RPC: Registered udp transport module. [ 0.907591] RPC: Registered tcp transport module. [ 0.909478] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.911797] NET: Registered protocol family 44 [ 0.913549] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.915891] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.918099] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.920378] PCI: CLS 0 bytes, default 64 [ 0.921866] Unpacking initramfs... [ 2.323001] debug: unmapping init [mem 0xffff8c40bcc54000-0xffff8c40bffbffff] [ 2.326669] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.328794] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.331515] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.837639] Initialise system trusted keyrings [ 2.839323] Key type blacklist registered [ 2.840996] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.850990] zbud: loaded [ 2.855359] *** VALIDATE nfs *** [ 2.856874] *** VALIDATE nfs4 *** [ 2.858608] pstore: using deflate compression [ 2.862446] Platform Keyring initialized [ 2.961622] NET: Registered protocol family 38 [ 2.963409] Key type asymmetric registered [ 2.965101] Asymmetric key parser 'x509' registered [ 2.966885] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.969609] io scheduler mq-deadline registered [ 2.971106] io scheduler kyber registered [ 2.972767] io scheduler bfq registered [ 2.974691] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.977957] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.981103] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.984307] ACPI: Power Button [PWRF] [ 2.990793] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.998859] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.034578] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.040453] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.055203] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.083352] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.110664] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.115228] Non-volatile memory driver v1.3 [ 3.116981] Linux agpgart interface v0.103 [ 3.147536] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.150749] vda: detected capacity change from 0 to 68624384 [ 3.182832] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.185895] vdb: detected capacity change from 0 to 1073741824 [ 3.202162] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.205413] vdc: detected capacity change from 0 to 2621440000 [ 3.226116] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.230883] vdd: detected capacity change from 0 to 2621440000 [ 3.251918] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.254212] vde: detected capacity change from 0 to 4294967296 [ 3.268623] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.270514] vdf: detected capacity change from 0 to 4294967296 [ 3.280590] libphy: Fixed MDIO Bus: probed [ 3.288667] usbcore: registered new interface driver usbserial_generic [ 3.291366] usbserial: USB Serial support registered for generic [ 3.293917] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.298584] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.300617] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.303165] mousedev: PS/2 mouse device common for all mice [ 3.306217] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.306634] rtc_cmos 00:05: RTC can wake from S4 [ 3.314505] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.316185] rtc_cmos 00:05: registered as rtc0 [ 3.320375] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.322413] intel_pstate: CPU model not supported [ 3.324210] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.325686] hid: raw HID events driver (C) Jiri Kosina [ 3.330205] usbcore: registered new interface driver usbhid [ 3.332689] usbhid: USB HID core driver [ 3.334741] drop_monitor: Initializing network drop monitor service [ 3.337419] Initializing XFRM netlink socket [ 3.339884] NET: Registered protocol family 10 [ 3.343656] Segment Routing with IPv6 [ 3.345397] NET: Registered protocol family 17 [ 3.348158] mpls_gso: MPLS GSO support [ 3.354339] RAS: Correctable Errors collector initialized. [ 3.356752] AVX version of gcm_enc/dec engaged. [ 3.358475] AES CTR mode by8 optimization enabled [ 3.438763] sched_clock: Marking stable (3438742961, 0)->(4367413796, -928670835) [ 3.441345] registered taskstats version 1 [ 3.443071] Loading compiled-in X.509 certificates [ 3.444938] zswap: loaded using pool lzo/zbud [ 3.471993] Key type big_key registered [ 3.484605] Key type encrypted registered [ 3.486329] ima: No TPM chip found, activating TPM-bypass! [ 3.488371] ima: Allocated hash algorithm: sha1 [ 3.490178] ima: No architecture policies found [ 3.492074] evm: Initialising EVM extended attributes: [ 3.494145] evm: security.selinux [ 3.495433] evm: security.ima [ 3.496709] evm: security.capability [ 3.498081] evm: HMAC attrs: 0x1 [ 3.500239] rtc_cmos 00:05: setting system clock to 2026-01-22 09:30:02 UTC (1769074202) [ 3.506795] debug: unmapping init [mem 0xffffffffa2803000-0xffffffffa29fffff] [ 3.509833] debug: unmapping init [mem 0xffffffffa1582000-0xffffffffa1858fff] [ 3.520206] Write protecting the kernel read-only data: 28672k [ 3.523838] debug: unmapping init [mem 0xffffffff9fc03000-0xffffffff9fdfffff] [ 3.527136] debug: unmapping init [mem 0xffffffffa0514000-0xffffffffa05fffff] [ 3.566110] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.574628] systemd[1]: Detected virtualization kvm. [ 3.576576] systemd[1]: Detected architecture x86-64. [ 3.578367] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.602975] systemd[1]: No hostname configured. [ 3.605717] systemd[1]: Set hostname to . [ 3.608226] random: systemd: uninitialized urandom read (16 bytes read) [ 3.610667] systemd[1]: Initializing machine ID from random generator. [ 3.744149] random: systemd: uninitialized urandom read (16 bytes read) [ 3.747175] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.751175] random: systemd: uninitialized urandom read (16 bytes read) [ 3.753147] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.756874] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... [ 3.876195] random: fast init done Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.281777] device-mapper: uevent: version 1.0.3 [ 4.283079] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.945131] virtio_net virtio0 ens2: renamed from eth0 [ 5.146250] scsi host0: ata_piix [ 5.191751] scsi host1: ata_piix [ 5.192970] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.194897] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.932956] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.003424] random: crng init done [ 10.004329] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.445951] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.606575] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.868549] SELinux: Disabled at runtime. [ 11.921690] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.927365] systemd[1]: Detected virtualization kvm. [ 11.928612] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.352197] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.354564] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.359126] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.362197] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.366168] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.374100] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.381429] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [[ 12.499253] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.882653] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.104115] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.150820] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.297684] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.311662] EDAC sbridge: Ver: 1.1.2 [ 14.947403] Key type dns_resolver registered [ 15.250040] NFS: Registering the id_resolver key type [ 15.252111] Key type id_resolver registered [ 15.253768] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 29.962048] hrtimer: interrupt took 4036550 ns [ 80.167993] libcfs: loading out-of-tree module taints kernel. [ 80.202477] Key type ._llcrypt registered [ 80.206168] Key type .llcrypt registered [ 80.314467] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 100.997821] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 102.931812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.952596] alg: No test for adler32 (adler32-zlib) [ 104.556870] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 105.843555] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 107.632079] Key type lgssc registered [ 110.074195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 133.208701] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.646382] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.961891] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.792781] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.332572] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.299651] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 196.705650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.776909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.795342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.178303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.255559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.416972] Lustre: lustre-MDT0000: new disk, initializing [ 198.611233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.646381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 203.688680] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.422593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 220.512617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 220.588509] Lustre: 6522:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.639386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.650144] Lustre: Skipped 1 previous similar message [ 220.772878] Lustre: lustre-MDT0001: new disk, initializing [ 220.910332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.992138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 221.004818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.046360] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.036875] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 243.524479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 243.621419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 243.895851] Lustre: lustre-OST0000: new disk, initializing [ 243.901212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 243.976732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.407964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 249.420842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 249.540442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 251.784212] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.795903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 268.916974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 269.014919] Lustre: lustre-OST0001: new disk, initializing [ 269.023420] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 269.103694] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.614037] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.018240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 277.035220] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 277.135754] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 292.032931] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.034880] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 312.935887] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 318.759823] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 324.586684] Lustre: DEBUG MARKER: Client: 2.17.50.3 [ 328.139975] Lustre: DEBUG MARKER: MDS: 2.17.50.3 [ 331.452855] Lustre: DEBUG MARKER: OSS: 2.17.50.3 [ 333.947686] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 22 04:35:30 EST 2026 [ 356.963147] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 358.818980] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 374.239747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.241034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.255450] Lustre: Skipped 3 previous similar messages [ 374.282602] Lustre: Skipped 3 previous similar messages [ 379.366803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.373153] Lustre: Skipped 3 previous similar messages [ 380.346411] LustreError: 12231:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 380.701493] Lustre: server umount lustre-MDT0000 complete [ 385.525980] LustreError: 12233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074584 with bad export cookie 6412415772028344029 [ 385.534161] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 385.539645] LustreError: 12233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 389.602733] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.604448] LustreError: 10427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.622579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 389.642068] LustreError: 10427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 389.663897] Lustre: Skipped 1 previous similar message [ 390.633711] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.657593] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 391.728302] LustreError: 12435:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 391.733826] LustreError: 12435:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 392.000489] Lustre: server umount lustre-MDT0001 complete [ 397.183055] LustreError: 12637:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 397.184989] LustreError: 12637:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 397.258275] Lustre: server umount lustre-OST0000 complete [ 402.165369] LustreError: 12838:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 402.171463] LustreError: 12838:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 402.425705] Lustre: server umount lustre-OST0001 complete [ 411.983857] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 428.148946] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 441.253216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 449.841639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 457.209275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 465.144584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 473.472303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 473.559682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.988776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 474.040751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 474.174926] Lustre: lustre-MDT0000: new disk, initializing [ 474.292351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.312811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 478.845397] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 490.447055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 490.499355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 490.568963] Lustre: 15043:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 490.622671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 490.625353] Lustre: Skipped 1 previous similar message [ 490.704779] Lustre: lustre-MDT0001: new disk, initializing [ 490.807231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 490.834608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 490.838499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 495.781381] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 503.991477] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 508.673776] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 515.944534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 516.003180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 516.235621] Lustre: lustre-OST0000: new disk, initializing [ 516.246492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 516.354419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.025953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 518.048067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 518.215649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 523.050266] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 532.050549] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 537.349334] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 537.650640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 542.625726] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 542.875942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 548.835644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.838352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.840777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.840785] Lustre: Skipped 2 previous similar messages [ 548.854654] Lustre: Skipped 2 previous similar messages [ 548.886187] LustreError: Skipped 1 previous similar message [ 551.294866] LustreError: 17984:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 551.297788] LustreError: 17984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 551.469967] Lustre: server umount lustre-OST0000 complete [ 557.541125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 557.554368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.580887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 557.588837] Lustre: Skipped 1 previous similar message [ 561.545359] LustreError: 18185:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 561.565400] LustreError: 18185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 561.747177] Lustre: server umount lustre-MDT0000 complete [ 564.195496] LustreError: 15052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.237644] LustreError: 15052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 566.487217] LustreError: 18187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074765 with bad export cookie 6412415772028345562 [ 566.499181] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.509897] LustreError: 18187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 567.043271] Lustre: server umount lustre-MDT0001 complete [ 573.636917] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:39:31 (1769074771) [ 595.113979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.626309] LustreError: 19525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.771688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.490741] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 601.058938] LustreError: 19525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.638490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 614.709691] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 621.669480] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 626.166341] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 628.419062] Lustre: Failing over lustre-MDT0000 [ 628.553941] LustreError: 20869:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 628.565769] LustreError: 20869:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 628.682308] Lustre: server umount lustre-MDT0000 complete [ 630.752218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.757886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.765347] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.783348] Lustre: Skipped 2 previous similar messages [ 630.808569] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 646.112178] Lustre: 3661:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769074829/real 1769074829] req@ffff8c400300b100 x1855008857563264/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769074845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 646.163569] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.757737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.234269] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.250401] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 656.740306] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 661.991658] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 661.995958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 662.027905] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 663.230866] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 670.835291] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 676.339112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.341556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 676.377450] Lustre: Skipped 1 previous similar message [ 676.398367] Lustre: Skipped 3 previous similar messages [ 681.759926] LustreError: 21853:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 681.769492] LustreError: 21853:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 682.057977] Lustre: server umount lustre-MDT0000 complete [ 684.513404] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.539313] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 685.925928] LustreError: 19505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074884 with bad export cookie 6412415772028346976 [ 685.939477] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.943857] LustreError: 19505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 686.376546] Lustre: server umount lustre-MDT0001 complete [ 695.704208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.277607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.285794] Lustre: Skipped 2 previous similar messages [ 700.818304] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 710.488741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 717.620779] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 725.572986] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 730.338789] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 732.517497] Lustre: Failing over lustre-MDT0000 [ 732.690159] Lustre: server umount lustre-MDT0000 complete [ 735.589083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.075059] LustreError: 23426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 751.075305] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769074934/real 1769074934] req@ffff8c413a098700 x1855008857593216/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769074950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 751.088816] LustreError: 23426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 751.146476] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.924442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.449060] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x58fd7a2b93dca603 [ 761.461167] Lustre: MGC192.168.201.118@tcp: Connection restored to 0@lo (at 0@lo) [ 761.464133] Lustre: Skipped 1 previous similar message [ 761.752710] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 766.145943] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 766.952642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 766.959460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 766.992361] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 773.663833] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 777.777466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.787211] Lustre: Skipped 1 previous similar message [ 777.792482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 777.804652] Lustre: Skipped 2 previous similar messages [ 783.969384] LustreError: 24857:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 783.975100] LustreError: 24857:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 784.127546] Lustre: server umount lustre-MDT0000 complete [ 788.327780] LustreError: 22512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074987 with bad export cookie 6412415772028347907 [ 788.344625] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 788.648541] Lustre: server umount lustre-MDT0001 complete [ 798.552693] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:43:16 (1769074996) [ 816.433398] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 827.486780] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 838.754996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 845.979925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.603490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 860.136343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 871.746425] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 883.882839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 883.970550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.209913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 884.261984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 884.390474] Lustre: lustre-MDT0000: new disk, initializing [ 884.537239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.544231] Lustre: Skipped 2 previous similar messages [ 884.563123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 889.268995] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 901.074948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 901.193975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 901.343242] Lustre: 28529:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 901.717561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 906.572960] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 915.966495] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 921.303491] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 929.968617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 930.143056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 932.139571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 932.209460] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 932.276570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 932.288818] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 936.965184] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 945.128378] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 951.312618] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 978.400370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 978.408780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.413118] LustreError: Skipped 1 previous similar message [ 978.414077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.414082] Lustre: Skipped 2 previous similar messages [ 978.445120] Lustre: Skipped 2 previous similar messages [ 980.283192] LustreError: 31194:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 980.291219] LustreError: 31194:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 980.461101] Lustre: server umount lustre-OST0000 complete [ 988.647583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 991.629540] Lustre: server umount lustre-MDT0000 complete [ 993.761275] LustreError: 28541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.781729] LustreError: 28541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 995.561480] LustreError: 28522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075194 with bad export cookie 6412415772028348313 [ 995.564452] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.569520] LustreError: 28522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 995.940789] Lustre: server umount lustre-MDT0001 complete [ 1009.499439] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1012.822032] Key type lgssc unregistered [ 1013.239346] LNet: 32130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.260594] LNetError: 32130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.335939] LNet: Removed LNI 192.168.201.118@tcp [ 1014.677321] Key type .llcrypt unregistered [ 1014.681307] Key type ._llcrypt unregistered [ 1044.344162] Key type ._llcrypt registered [ 1044.345886] Key type .llcrypt registered [ 1044.440824] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1045.520959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1045.749045] alg: No test for adler32 (adler32-zlib) [ 1046.843306] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1047.103733] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1048.841819] Key type lgssc registered [ 1050.137598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1057.856528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1057.880377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.405995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.509724] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1069.556717] LustreError: 33543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.579550] LustreError: 33543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1073.998642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1074.307568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.329701] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1086.412664] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1090.828626] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1098.248729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1098.586780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.656369] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1104.801410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1105.327413] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1111.708819] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1128.609213] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:48:46 (1769075326) [ 1131.251600] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1133.602284] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1142.016864] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:48:59 (1769075339) [ 1151.411352] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:49:09 (1769075349) [ 1152.494494] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1162.198087] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1162.205364] Lustre: Skipped 2 previous similar messages [ 1163.956439] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:49:21 (1769075361) [ 1166.454155] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1166.469369] Lustre: Skipped 2 previous similar messages [ 1175.718168] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1175.736604] Lustre: Skipped 1 previous similar message [ 1192.788454] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1192.793737] Lustre: Skipped 5 previous similar messages [ 1196.372594] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:49:53 (1769075393) [ 1199.580856] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1203.459925] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1214.803891] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:50:12 (1769075412) [ 1231.613165] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1231.618424] Lustre: Skipped 7 previous similar messages [ 1233.964839] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:50:31 (1769075431) [ 1258.465374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1258.468141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.490829] LustreError: Skipped 1 previous similar message [ 1258.503504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.509639] Lustre: Skipped 1 previous similar message [ 1258.559016] Lustre: Skipped 1 previous similar message [ 1259.507286] LustreError: 40117:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1259.791965] Lustre: server umount lustre-OST0000 complete [ 1268.710607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.711867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.740304] Lustre: Skipped 1 previous similar message [ 1269.922128] LustreError: 40320:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1269.937978] LustreError: 40320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1270.170964] Lustre: server umount lustre-MDT0000 complete [ 1273.825019] LustreError: 33543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.838984] LustreError: 33543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1274.767979] LustreError: 33524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075473 with bad export cookie 14097154868037713621 [ 1274.783598] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.962283] LustreError: 40522:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1274.966277] LustreError: 40522:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1275.079315] Lustre: server umount lustre-MDT0001 complete [ 1287.099155] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1289.501481] Key type lgssc unregistered [ 1289.764983] LNet: 41053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.777210] LNetError: 41053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.807772] LNet: Removed LNI 192.168.201.118@tcp [ 1290.482272] Key type .llcrypt unregistered [ 1290.484218] Key type ._llcrypt unregistered [ 1312.506466] Key type ._llcrypt registered [ 1312.511323] Key type .llcrypt registered [ 1312.676237] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1331.968871] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1333.511476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1333.633422] alg: No test for adler32 (adler32-zlib) [ 1334.875818] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1335.155240] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1336.847185] Key type lgssc registered [ 1338.750392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1349.924925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1358.303580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1365.655102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1373.553793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1381.312646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1381.420916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.442611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.817228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1382.866944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1382.980898] Lustre: lustre-MDT0000: new disk, initializing [ 1383.179200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.228252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1388.860852] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1402.239759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1402.366375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1402.452650] Lustre: 44187:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1402.502537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1402.505807] Lustre: Skipped 1 previous similar message [ 1402.623718] Lustre: lustre-MDT0001: new disk, initializing [ 1402.752849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1402.865509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1402.895449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1409.412072] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1419.062585] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1423.747596] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1432.154707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1432.214743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1432.524749] Lustre: lustre-OST0000: new disk, initializing [ 1432.544707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1432.729178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.843850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1438.864554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1438.992258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1440.364633] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1450.732910] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1457.032286] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1457.378520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1463.320174] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1463.732019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1469.920470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1469.925641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.930017] LustreError: Skipped 1 previous similar message [ 1469.930716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1469.961029] Lustre: Skipped 1 previous similar message [ 1472.391113] LustreError: 47131:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1472.708866] Lustre: server umount lustre-OST0000 complete [ 1479.648518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1479.656400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.688277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.694168] Lustre: Skipped 1 previous similar message [ 1483.676735] LustreError: 47332:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1483.679034] LustreError: 47332:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1483.824637] Lustre: server umount lustre-MDT0000 complete [ 1488.166635] LustreError: 44180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075687 with bad export cookie 5399782707630923094 [ 1488.169649] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1488.186515] LustreError: 44180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1488.442842] LustreError: 47534:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1488.459926] LustreError: 47534:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1488.704123] Lustre: server umount lustre-MDT0001 complete [ 1500.797942] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:54:57 (1769075697) [ 1508.180394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.588798] LustreError: 48253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.609292] LustreError: 48253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1508.711440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.283376] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1513.971164] LustreError: 48253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1518.054276] LustreError: 48253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1523.172548] LustreError: 48253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1523.952092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1524.197338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1529.438575] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1536.420407] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1540.953387] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1549.501554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1549.738523] LustreError: 49855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.115394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1558.625625] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1566.286508] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1572.839261] LustreError: 50452:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1574.010640] Lustre: Modifying parameter general.timeout=1 in log params [ 1575.092142] Lustre: Modifying parameter general.timeout=2 in log params [ 1577.281133] Lustre: Modifying parameter general.timeout=4 in log params [ 1577.285677] Lustre: Skipped 1 previous similar message [ 1582.026184] Lustre: Modifying parameter general.timeout=8 in log params [ 1582.030350] Lustre: Skipped 3 previous similar messages [ 1590.149613] Lustre: Modifying parameter general.timeout=15 in log params [ 1590.156887] Lustre: Skipped 6 previous similar messages [ 1606.569675] Lustre: Modifying parameter general.timeout=30 in log params [ 1606.579531] Lustre: Skipped 14 previous similar messages [ 1638.975227] Lustre: Modifying parameter general.timeout=62 in log params [ 1638.980911] Lustre: Skipped 31 previous similar messages [ 1703.765919] Lustre: Modifying parameter general.timeout=120 in log params [ 1703.773969] Lustre: Skipped 57 previous similar messages [ 1726.132958] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:58:43 (1769075923) [ 1729.571869] Lustre: 57409:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1731.810741] Lustre: 57521:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1733.931338] Lustre: 57619:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1739.727845] Lustre: 57927:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1739.737581] Lustre: 57927:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1770.281589] Lustre: 59289:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1770.315333] LustreError: 59289:0:(mgs_handler.c:1129:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1775.930572] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:59:32 (1769075972) [ 1792.488220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.494022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1792.512583] Lustre: Skipped 2 previous similar messages [ 1792.541222] Lustre: Skipped 4 previous similar messages [ 1797.172469] LustreError: 59990:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1797.181466] LustreError: 59990:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1797.397070] Lustre: server umount lustre-MDT0000 complete [ 1797.601456] LustreError: 48252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.622724] LustreError: 48252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1802.554899] LustreError: 48234:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076001 with bad export cookie 5399782707630923976 [ 1802.570021] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.722650] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.742468] Lustre: Skipped 1 previous similar message [ 1802.748677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1802.846083] LustreError: 60191:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1802.861350] LustreError: 60191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1803.129477] Lustre: server umount lustre-MDT0001 complete [ 1808.309440] Lustre: server umount lustre-OST0000 complete [ 1818.067447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1822.915313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1827.770415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1832.325602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1840.047787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.382878] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1840.465374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1840.482333] Lustre: Skipped 32 previous similar messages [ 1840.867323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.476938] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1854.132763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1854.298659] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1854.371616] Lustre: 61844:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1854.512424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1859.735837] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1866.606995] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1870.880797] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1877.818309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1878.061057] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1878.312837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1878.315923] Lustre: Skipped 1 previous similar message [ 1885.746906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1892.252366] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1893.888416] Lustre: 64092:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1905.639612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1905.650093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.664458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1907.394513] LustreError: 64338:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1907.396662] LustreError: 64338:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1907.589808] Lustre: server umount lustre-OST0000 complete [ 1916.385715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.387214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1916.414829] Lustre: Skipped 2 previous similar messages [ 1918.244719] Lustre: server umount lustre-MDT0000 complete [ 1921.506845] LustreError: 61855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.532368] LustreError: 61855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1922.912107] LustreError: 61837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076121 with bad export cookie 5399782707630927189 [ 1922.916073] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.920136] LustreError: 61837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1923.446152] Lustre: server umount lustre-MDT0001 complete [ 1933.580616] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1936.290326] Key type lgssc unregistered [ 1936.590295] LNet: 65281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.600735] LNetError: 65281:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.626123] LNet: Removed LNI 192.168.201.118@tcp [ 1937.272285] Key type .llcrypt unregistered [ 1937.275476] Key type ._llcrypt unregistered [ 1963.257907] Key type ._llcrypt registered [ 1963.263052] Key type .llcrypt registered [ 1963.383520] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1964.717194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.754544] alg: No test for adler32 (adler32-zlib) [ 1965.882647] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1966.081781] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1967.751367] Key type lgssc registered [ 1969.123911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1978.101570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1978.115848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.586686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.386563] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1989.155530] LustreError: 66687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.179572] LustreError: 66687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1994.214603] LustreError: 66687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.351605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1995.011907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1999.905277] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2006.410405] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2012.056080] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2020.948968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2021.664752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.979640] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2036.115882] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2049.685279] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2064.875774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.884985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2064.899065] Lustre: Skipped 2 previous similar messages [ 2064.909340] Lustre: Skipped 2 previous similar messages [ 2068.916378] LustreError: 69402:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2069.068904] Lustre: server umount lustre-MDT0000 complete [ 2069.989280] LustreError: 67667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.005106] LustreError: 67667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2073.545356] LustreError: 67413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076272 with bad export cookie 2087075431591784839 [ 2073.559902] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.562520] LustreError: 67413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2073.702182] LustreError: 69604:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2073.707832] LustreError: 69604:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2073.990277] Lustre: server umount lustre-MDT0001 complete [ 2089.530949] LustreError: 69805:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2089.536402] LustreError: 69805:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2089.657606] Lustre: server umount lustre-OST0000 complete [ 2098.461496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2103.179635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2107.661042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2112.393567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2122.273139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.877975] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2122.911624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2123.155442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.308476] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2140.383209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2140.550865] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2140.605629] Lustre: 71259:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2141.390752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2147.185553] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2154.545899] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2159.270572] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2168.601323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2168.908232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2169.605820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.397625] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2184.735688] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2186.644936] Lustre: 73501:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2188.854986] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2199.527146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.529365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2199.546467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2201.775855] LustreError: 73759:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2201.783303] LustreError: 73759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2202.240159] Lustre: server umount lustre-OST0000 complete [ 2207.712357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2207.715158] LustreError: Skipped 1 previous similar message [ 2207.722114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.732899] Lustre: Skipped 1 previous similar message [ 2207.749719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.755866] Lustre: Skipped 1 previous similar message [ 2209.764899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.771983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2209.821870] Lustre: Skipped 1 previous similar message [ 2213.374220] LustreError: 73963:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2213.383647] LustreError: 73963:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2214.031670] Lustre: server umount lustre-MDT0000 complete [ 2214.886143] LustreError: 71269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.926352] LustreError: 71269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2219.632483] LustreError: 71251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076418 with bad export cookie 2087075431591785686 [ 2219.650763] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2219.659283] LustreError: 71251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2220.386448] Lustre: server umount lustre-MDT0001 complete [ 2235.800651] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2238.526076] Key type lgssc unregistered [ 2238.883908] LNet: 74697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.899444] LNetError: 74697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.929570] LNet: Removed LNI 192.168.201.118@tcp [ 2239.880124] Key type .llcrypt unregistered [ 2239.881628] Key type ._llcrypt unregistered [ 2273.126448] Key type ._llcrypt registered [ 2273.131889] Key type .llcrypt registered [ 2273.369181] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2275.125923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2275.266982] alg: No test for adler32 (adler32-zlib) [ 2276.469954] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2276.672537] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2278.335197] Key type lgssc registered [ 2279.988158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.681359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.706110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.494838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.470085] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2301.921557] LustreError: 76097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.931035] LustreError: 76097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2307.046777] LustreError: 76097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.935940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2308.654251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2313.808792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2323.896757] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2328.801455] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2337.809955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2338.609061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.944396] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2354.986643] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2368.564047] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2369.691934] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2370.713247] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2372.810489] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2372.816283] Lustre: Skipped 1 previous similar message [ 2377.956367] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2377.959820] Lustre: Skipped 3 previous similar messages [ 2386.045373] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2386.049859] Lustre: Skipped 6 previous similar messages [ 2402.567071] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2402.581334] Lustre: Skipped 13 previous similar messages [ 2435.040944] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2435.052208] Lustre: Skipped 27 previous similar messages [ 2493.037775] Lustre: 83905:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2496.490340] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:11:33 (1769076693) [ 2498.257069] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2500.610972] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:11:37 (1769076697) [ 2507.747530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.765075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.790827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.802644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.803748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.824657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2512.396216] LustreError: 84201:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2512.603700] Lustre: server umount lustre-OST0000 complete [ 2518.506929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2518.521082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2518.550175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2518.553029] Lustre: Skipped 1 previous similar message [ 2524.354900] LustreError: 84402:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2524.372675] LustreError: 84402:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2524.779523] Lustre: server umount lustre-MDT0000 complete [ 2525.154486] LustreError: 76814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.322890] LustreError: 76077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076728 with bad export cookie 8136356619736022262 [ 2529.332335] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.336864] LustreError: 76077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2529.483795] LustreError: 84604:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2529.489579] LustreError: 84604:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2529.747849] Lustre: server umount lustre-MDT0001 complete [ 2542.852530] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2546.358795] Key type lgssc unregistered [ 2546.572361] LNet: 85136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.577509] LNetError: 85136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.593761] LNet: Removed LNI 192.168.201.118@tcp [ 2547.752843] Key type .llcrypt unregistered [ 2547.754444] Key type ._llcrypt unregistered [ 2554.936607] Key type ._llcrypt registered [ 2554.939231] Key type .llcrypt registered [ 2555.174525] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2560.469268] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules [ 2561.221166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2561.239155] alg: No test for adler32 (adler32-zlib) [ 2562.317033] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2562.512343] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2564.200355] Key type lgssc registered [ 2565.710891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2567.362874] LustreError: 86304:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 2587.022446] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2590.090924] LustreError: 86304:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2598.022536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2598.044967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2599.917890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.562774] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2609.122641] LustreError: 87292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.164281] LustreError: 87292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2614.250983] LustreError: 87292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.376890] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:13:35 (1769076815) [ 2619.373549] LustreError: 87292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.481747] LustreError: 87292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.710177] LustreError: 87324:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2624.722854] LustreError: 87324:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2624.729306] LustreError: 87324:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 2624.901198] LustreError: 88099:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2625.225992] Lustre: server umount lustre-MDT0000 complete [ 2643.691740] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2647.331580] Key type lgssc unregistered [ 2647.611354] LNet: 88784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2647.614573] LNetError: 88784:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2647.632846] LNet: Removed LNI 192.168.201.118@tcp [ 2648.789757] Key type .llcrypt unregistered [ 2648.794422] Key type ._llcrypt unregistered [ 2672.515486] Key type ._llcrypt registered [ 2672.521282] Key type .llcrypt registered [ 2672.614664] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2673.990038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.005374] alg: No test for adler32 (adler32-zlib) [ 2675.323306] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2675.712205] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2677.447193] Key type lgssc registered [ 2679.174617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2689.028284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2689.080282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.680981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.843243] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2700.265177] LustreError: 89999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.313112] LustreError: 89999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2705.391962] LustreError: 89999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.423887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2707.390634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2712.425548] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2719.951795] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2724.415241] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2734.542706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2735.573491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2744.612206] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2751.966523] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2811.956708] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:16:48 (1769077008) [ 2815.126766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2817.979311] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:16:54 (1769077014) [ 2824.686456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2824.705155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.726904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.769600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.781716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.787085] Lustre: Skipped 2 previous similar messages [ 2830.225300] LustreError: 92929:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2830.458865] Lustre: server umount lustre-MDT0000 complete [ 2833.890868] LustreError: 89999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.923707] LustreError: 89999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2834.977223] LustreError: 89980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077033 with bad export cookie 7658716025536695396 [ 2835.001313] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.213141] LustreError: 93130:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2835.223764] LustreError: 93130:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2835.676718] Lustre: server umount lustre-MDT0001 complete [ 2850.255900] LustreError: 93333:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2850.259137] LustreError: 93333:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2850.771494] Lustre: server umount lustre-OST0000 complete [ 2866.047139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.634676] LustreError: 93965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.878123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.290530] LustreError: 93965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.709402] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2881.505435] LustreError: 93964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.524646] LustreError: 93964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2883.727800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2884.673173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2890.534605] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2898.668962] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2904.557537] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2912.718110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2920.404910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2920.491564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2920.688762] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2920.695603] LustreError: 93954:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2920.721743] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2920.725991] LustreError: 95828:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2920.733879] LustreError: 95828:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2920.740114] LustreError: 95828:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2920.804855] Lustre: server umount lustre-OST0000 complete [ 2920.810455] LustreError: 95828:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2930.837460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2931.240814] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2931.253238] LustreError: 93953:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2931.308970] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2931.318094] LustreError: 96296:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2931.325222] LustreError: 96296:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2931.330926] LustreError: 96296:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2931.474782] Lustre: server umount lustre-OST0000 complete [ 2931.476862] LustreError: 96296:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2935.290393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.008850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2943.278433] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2943.280778] Lustre: Found index 0 for lustre-OST0000, updating log [ 2943.296385] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2943.379282] Lustre: lustre-OST0000: new disk, initializing [ 2943.382280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2943.763567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.972363] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2956.374605] Lustre: Failing over lustre-OST0000 [ 2956.426087] LustreError: 97386:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2956.433105] LustreError: 97386:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2956.586415] Lustre: server umount lustre-OST0000 complete [ 2964.472655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.485947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.490852] Lustre: Skipped 2 previous similar messages [ 2964.502935] Lustre: Skipped 1 previous similar message [ 2967.218633] LustreError: 97587:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2967.229258] LustreError: 97587:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2967.473912] Lustre: server umount lustre-MDT0000 complete [ 2969.569753] LustreError: 94682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.609632] LustreError: 94682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2972.244122] LustreError: 93947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077171 with bad export cookie 7658716025536698140 [ 2972.255256] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.258309] LustreError: 93947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2982.952420] LustreError: 97789:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2982.962916] LustreError: 97789:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2983.535397] Lustre: server umount lustre-MDT0001 complete [ 2997.829710] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:19:54 (1769077194) [ 3019.011159] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3031.769430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.091295] LustreError: 99160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.247273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.212859] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3048.463633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3055.529550] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3059.334491] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3067.701287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3068.808268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.814650] Lustre: Skipped 1 previous similar message [ 3068.839217] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3073.897043] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3073.919783] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3073.919939] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3076.570533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3087.787821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3087.888866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3088.000555] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3088.089941] Lustre: lustre-OST0001: new disk, initializing [ 3088.096725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3093.602264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3093.611798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3093.722875] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3097.238763] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3107.526325] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3115.976180] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3126.248300] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3126.256242] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.268266] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3131.411906] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 3131.424241] Lustre: Skipped 4 previous similar messages [ 3132.020679] LustreError: 102413:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3132.024596] LustreError: 102413:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3132.179600] Lustre: server umount lustre-MDT0001 complete [ 3134.958736] LustreError: 99160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.971340] LustreError: 99160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3136.973567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3147.970447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3148.157722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3148.161849] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3148.210990] Lustre: 99149:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3149.381274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3149.383911] Lustre: Skipped 1 previous similar message [ 3149.736599] LustreError: 102972:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3149.755095] LustreError: 102972:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3149.774076] LustreError: 99152:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3149.934610] LustreError: 102985:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -17 [ 3149.939922] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3149.939922] [ 3154.440781] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3154.690737] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3169.581385] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 05:22:46 (1769077366) [ 3191.382732] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3212.791610] Lustre: 105029:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3242.971141] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3247.772142] Lustre: 106167:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3353.573580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3353.582547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.599781] Lustre: Skipped 2 previous similar messages [ 3353.606526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.652553] LustreError: 106647:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3359.658480] LustreError: 106647:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3359.749425] Lustre: server umount lustre-MDT0000 complete [ 3362.283657] LustreError: 99154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.309375] LustreError: 99154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 3364.171209] LustreError: 102933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077563 with bad export cookie 7658716025536699134 [ 3364.172292] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.179996] LustreError: 102933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3364.513811] Lustre: server umount lustre-MDT0001 complete [ 3379.519612] Lustre: server umount lustre-OST0000 complete [ 3383.327142] Lustre: 89309:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769077566/real 1769077566] req@ffff8c414086dc00 x1855011552632576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769077582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3383.341504] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.349285] Lustre: Skipped 3 previous similar messages [ 3383.721690] Lustre: server umount lustre-OST0001 complete [ 3390.832144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3402.111215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3403.331282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3413.875758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3424.917043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3426.593283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3451.896336] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3466.938464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.984687] Lustre: lustre-MDT0000: reset Object Index mappings [ 3467.458538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.158660] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3486.209830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3486.236663] Lustre: lustre-MDT0001: reset Object Index mappings [ 3492.593483] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3498.362589] Lustre: 111250:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3509.625493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3512.191960] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3513.212984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3520.648679] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3532.518787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3534.825701] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3534.877063] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3542.206447] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3552.932500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3558.309160] Lustre: 113107:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3630.166140] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:30:27 (1769077827) [ 3655.648883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3655.656692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.684910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.703498] Lustre: Skipped 4 previous similar messages [ 3659.486257] LustreError: 113464:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3659.494735] LustreError: 113464:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3661.727197] Lustre: server umount lustre-MDT0000 complete [ 3661.803532] LustreError: 112829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.833385] LustreError: 112829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 3665.784286] LustreError: 110113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077864 with bad export cookie 7658716025536746027 [ 3665.798747] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.813742] LustreError: 110113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3680.223540] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3680.588345] Lustre: server umount lustre-MDT0001 complete [ 3692.546361] Lustre: server umount lustre-OST0000 complete [ 3704.050851] Lustre: server umount lustre-OST0001 complete [ 3712.855665] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3724.352776] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3735.736808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3745.224883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3754.203872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3762.187958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3770.499641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3770.614893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.971036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3771.022895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3771.216842] Lustre: lustre-MDT0000: new disk, initializing [ 3771.388668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.395019] Lustre: Skipped 3 previous similar messages [ 3771.419543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3778.083157] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3786.223605] Lustre: server umount lustre-MDT0000 complete [ 3788.944176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3800.427651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3800.539566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.190240] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3801.221624] Lustre: 117350:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3801.250481] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3807.725526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3815.374355] LustreError: 117864:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3815.381971] LustreError: 117864:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3815.504900] Lustre: server umount lustre-MDT0000 complete [ 3825.431204] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:33:43 (1769078023) [ 3827.190139] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3829.749289] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:33:46 (1769078026) [ 3851.043961] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3862.085221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3868.808230] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3881.795184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3881.899587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3882.097363] Lustre: 119370:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3882.218113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3882.229778] Lustre: Skipped 1 previous similar message [ 3882.396660] Lustre: lustre-MDT0001: new disk, initializing [ 3882.576983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3882.593620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3887.801389] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3893.013206] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3901.347649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3901.429284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3901.708408] Lustre: lustre-OST0000: new disk, initializing [ 3901.711873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3903.621766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3903.630188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3903.763541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3909.513753] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3920.658588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3920.784656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3921.010606] Lustre: lustre-OST0001: new disk, initializing [ 3921.016984] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3922.323902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3922.340548] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3922.468612] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3927.484756] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3937.601408] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3941.925674] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4407.707253] Lustre: 128193:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4407.716877] Lustre: 128193:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4407.725172] Lustre: 128193:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4407.731377] Lustre: 128193:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4407.739870] Lustre: 128193:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4407.746460] Lustre: 128193:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4686.924402] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:48:03 (1769078883) [ 4705.777344] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.783950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4705.787890] Lustre: Skipped 7 previous similar messages [ 4705.807573] Lustre: Skipped 15 previous similar messages [ 4708.905556] LustreError: 128425:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4708.916362] LustreError: 128425:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4709.470958] Lustre: server umount lustre-MDT0000 complete [ 4710.890853] LustreError: 120914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.918343] LustreError: 120914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4714.186181] LustreError: 120921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078913 with bad export cookie 7658716025536816426 [ 4714.188582] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4714.205774] LustreError: 120921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4716.001164] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.012733] Lustre: Skipped 1 previous similar message [ 4716.017235] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4716.034184] Lustre: Skipped 1 previous similar message [ 4720.962588] Lustre: server umount lustre-MDT0001 complete [ 4725.983599] Lustre: server umount lustre-OST0000 complete [ 4737.389834] Lustre: server umount lustre-OST0001 complete [ 4745.012314] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4759.111651] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4775.669594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4783.644772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4792.071340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4801.385704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4810.709328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4810.811372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4811.094656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4811.192147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4811.383364] Lustre: lustre-MDT0000: new disk, initializing [ 4811.554336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.569123] Lustre: Skipped 5 previous similar messages [ 4811.627868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4816.079652] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4830.832320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.906893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4830.972277] Lustre: 131222:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4831.043613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4831.045928] Lustre: Skipped 1 previous similar message [ 4831.119951] Lustre: lustre-MDT0001: new disk, initializing [ 4831.293397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4831.299745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4837.041490] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4847.328817] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4852.937395] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4862.920710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4863.052491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4863.377644] Lustre: lustre-OST0000: new disk, initializing [ 4863.391352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4864.679837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4864.688193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4864.843165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4871.063897] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4879.849399] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4892.022823] Lustre: lustre-MDD0000: changelog on [ 4895.485667] Lustre: lustre-MDD0001: changelog on [ 6654.346942] Lustre: lustre-MDD0001: changelog off [ 6655.587530] Lustre: 131230:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6656.183385] Lustre: lustre-MDD0000: changelog off [ 6662.117617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6662.118097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.121798] LustreError: Skipped 1 previous similar message [ 6662.122847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6662.129495] Lustre: Skipped 1 previous similar message [ 6662.137990] Lustre: Skipped 4 previous similar messages [ 6665.850238] LustreError: 141040:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6665.853435] LustreError: 141040:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6665.944448] Lustre: server umount lustre-OST0000 complete [ 6669.281100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6669.288961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.296316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.353882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.357918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.365883] Lustre: Skipped 1 previous similar message [ 6674.595334] LustreError: 141241:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6674.601286] LustreError: 141241:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6674.723348] Lustre: server umount lustre-MDT0000 complete [ 6677.305371] LustreError: 131216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080876 with bad export cookie 7658716025537595106 [ 6677.312729] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6677.317316] LustreError: 131216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6677.556075] Lustre: server umount lustre-MDT0001 complete [ 6683.999873] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6685.613138] Key type lgssc unregistered [ 6685.774820] LNet: 141974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6685.781380] LNetError: 141974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6685.793600] LNet: Removed LNI 192.168.201.118@tcp [ 6686.317354] Key type .llcrypt unregistered [ 6686.319261] Key type ._llcrypt unregistered [ 6698.943762] Key type ._llcrypt registered [ 6698.945191] Key type .llcrypt registered [ 6699.006495] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6709.557229] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6710.331038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6710.348306] alg: No test for adler32 (adler32-zlib) [ 6711.325394] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 6711.469860] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6713.079142] Key type lgssc registered [ 6713.896667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6719.297147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6723.516811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6727.687541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6732.251469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6736.215608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6736.249670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6736.256505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.376993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6737.394736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6737.442194] Lustre: lustre-MDT0000: new disk, initializing [ 6737.501610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.523610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6739.733833] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6746.280080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6746.317911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6746.361464] Lustre: 145134:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6746.408582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6746.411971] Lustre: Skipped 1 previous similar message [ 6746.472292] Lustre: lustre-MDT0001: new disk, initializing [ 6746.526514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6746.554096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6746.558281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6748.541752] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6753.393131] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6755.298504] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6758.908549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6758.939828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6759.069237] Lustre: lustre-OST0000: new disk, initializing [ 6759.072355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6759.111444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6762.176410] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6765.587603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6765.592025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6765.653606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6766.898900] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6770.101443] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module obdecho/obdecho [ 6771.200710] LustreError: 147756:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6771.209809] LustreError: 147756:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 6772.228498] LustreError: 147852:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 6780.902456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.910859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6780.911313] Lustre: Skipped 2 previous similar messages [ 6780.917073] Lustre: Skipped 2 previous similar messages [ 6784.461600] LustreError: 148192:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6784.503863] Lustre: server umount lustre-MDT0000 complete [ 6786.523844] LustreError: 145128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080985 with bad export cookie 12036422364266809276 [ 6786.526039] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6786.529920] LustreError: 145128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6786.598963] LustreError: 148395:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6786.600809] LustreError: 148395:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6786.676598] Lustre: server umount lustre-MDT0001 complete [ 6798.652040] LustreError: 148595:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6798.654876] LustreError: 148595:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6798.739884] Lustre: server umount lustre-OST0000 complete [ 6811.032858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6816.212833] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6821.115538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6824.889902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6828.720818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6832.771308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6837.194698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6837.225405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.364785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6837.379786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6837.422608] Lustre: lustre-MDT0000: new disk, initializing [ 6837.474677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6837.484334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6839.632779] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6846.270849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6846.309764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6846.357847] Lustre: 151556:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6846.400163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6846.403492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6846.406465] Lustre: Skipped 1 previous similar message [ 6846.446092] Lustre: lustre-MDT0001: new disk, initializing [ 6846.508662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6846.540347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6846.550163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6848.723828] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6854.813140] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6858.574743] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6869.880463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6869.995613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6870.449945] Lustre: lustre-OST0000: new disk, initializing [ 6870.478845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6870.639171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6877.992060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6878.021420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6878.311918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6880.065905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6891.092652] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6898.153598] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6898.424288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6904.318798] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6904.803337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6908.899959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.920343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6908.929601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6908.933804] Lustre: Skipped 1 previous similar message [ 6914.039613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6914.048118] Lustre: Skipped 1 previous similar message [ 6919.138478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6919.140478] Lustre: Skipped 1 previous similar message [ 6921.183134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6921.252234] LustreError: 154497:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6921.259479] LustreError: 154497:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6921.540229] Lustre: server umount lustre-OST0000 complete [ 6928.355582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6928.359270] LustreError: Skipped 1 previous similar message [ 6928.361794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6928.371673] Lustre: Skipped 1 previous similar message [ 6928.375906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6931.452739] LustreError: 154699:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6931.458639] LustreError: 154699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6931.581073] Lustre: server umount lustre-MDT0000 complete [ 6934.496974] LustreError: 151563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6934.511685] LustreError: 151563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6935.206222] LustreError: 151546:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081134 with bad export cookie 12036422364266810235 [ 6935.208049] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6935.214399] LustreError: 151546:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6935.465276] Lustre: server umount lustre-MDT0001 complete [ 6938.778956] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:25:36 (1769081136) [ 6940.556740] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6942.151491] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:25:39 (1769081139) [ 6949.316629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6949.670702] LustreError: 155573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6949.723393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6953.684257] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6954.977960] LustreError: 155573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.073173] LustreError: 155573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6961.239909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6965.022542] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6969.550150] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6972.521299] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6978.209280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6983.299517] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6987.928889] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7007.713202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7007.726143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7007.733960] Lustre: Skipped 1 previous similar message [ 7007.739825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7007.744146] Lustre: Skipped 2 previous similar messages [ 7010.957131] LustreError: 157837:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7010.962708] LustreError: 157837:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7011.020082] Lustre: server umount lustre-MDT0000 complete [ 7014.107363] LustreError: 155552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081213 with bad export cookie 12036422364266811117 [ 7014.115898] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7014.122643] LustreError: 155552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7014.419194] LustreError: 155569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.395122] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7015.410389] Lustre: Skipped 2 previous similar messages [ 7020.536712] Lustre: server umount lustre-MDT0001 complete [ 7026.415696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7037.018695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7037.217491] LustreError: 159102:0:(llog.c:1643:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7037.223882] Lustre: 159102:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7044.065363] LustreError: 159110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.089562] LustreError: 159110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7060.505056] LustreError: 159110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7060.520919] LustreError: 159110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 7094.754717] LustreError: 159109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.771623] LustreError: 159109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 7098.335555] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7098.344362] Lustre: 159102:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7125.983431] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7125.986330] Lustre: 155564:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7139.350346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7139.363841] Lustre: Skipped 2 previous similar messages [ 7144.350774] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7144.420373] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7146.463241] LustreError: 155564:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7151.814591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.777781] LustreError: 159110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7159.782256] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa709f65e5cb0e838 [ 7159.789084] LustreError: 159110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 7159.802501] LustreError: 159839:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.201.118@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000075a7c3df) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7159.810790] Lustre: MGC192.168.201.118@tcp: Connection restored to 0@lo (at 0@lo) [ 7163.785903] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7165.428465] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7169.584431] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7171.456553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7175.746636] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7177.883405] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7199.867982] LustreError: 160607:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7199.872310] LustreError: 160607:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7200.109882] Lustre: server umount lustre-MDT0000 complete [ 7201.253244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7201.254683] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.272605] Lustre: Skipped 2 previous similar messages [ 7203.375796] LustreError: 155553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081402 with bad export cookie 12036422364266817592 [ 7203.380161] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7203.384736] LustreError: 155553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7203.652778] Lustre: server umount lustre-MDT0001 complete [ 7217.067350] Lustre: server umount lustre-OST0000 complete [ 7236.314366] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7244.307062] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7252.620416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7258.699215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7264.785848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7270.579731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7277.359250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7277.436953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.745468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7277.772742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7277.862532] Lustre: lustre-MDT0000: new disk, initializing [ 7277.984274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7277.989072] Lustre: Skipped 1 previous similar message [ 7278.015750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7281.999528] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7291.411249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7291.490510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7291.586070] Lustre: 163959:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7291.827040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7295.872130] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7302.615903] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7306.508814] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7312.785466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7312.857125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7313.116973] Lustre: lustre-OST0000: new disk, initializing [ 7313.124535] Lustre: Skipped 1 previous similar message [ 7313.136558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7313.144530] Lustre: Skipped 2 previous similar messages [ 7318.616783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7318.622970] Lustre: Skipped 1 previous similar message [ 7318.630300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7318.721698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7318.850371] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7326.321809] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7330.541874] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7330.710768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7334.220448] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7334.379953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7339.489313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7339.492057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.498125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7339.498134] Lustre: Skipped 8 previous similar messages [ 7339.524545] Lustre: Skipped 1 previous similar message [ 7350.240381] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7350.283125] LustreError: 166897:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7350.287358] LustreError: 166897:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7350.436250] Lustre: server umount lustre-OST0000 complete [ 7368.167437] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7368.500561] Lustre: server umount lustre-MDT0000 complete [ 7370.208397] LustreError: 163972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7370.224838] LustreError: 163972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7372.030497] LustreError: 163954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081571 with bad export cookie 12036422364266834420 [ 7372.043355] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7372.046815] LustreError: 163954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7372.482577] Lustre: server umount lustre-MDT0001 complete [ 7380.302871] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:32:58 (1769081578) [ 7386.591960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.573629] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7398.228543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7402.961450] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7407.981540] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7411.040340] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7416.325241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7416.542499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7416.545103] Lustre: Skipped 4 previous similar messages [ 7420.280544] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7424.096732] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7441.949841] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:33:59 (1769081639) [ 7448.546724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.547835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7448.549278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7448.549283] Lustre: Skipped 11 previous similar messages [ 7448.553653] Lustre: Skipped 3 previous similar messages [ 7450.565579] LustreError: 170441:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7450.572145] LustreError: 170441:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7450.671964] Lustre: server umount lustre-OST0000 complete [ 7459.400893] Lustre: server umount lustre-MDT0000 complete [ 7461.963605] LustreError: 167957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081660 with bad export cookie 12036422364266835295 [ 7461.965367] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7461.972074] LustreError: 167957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7462.236700] Lustre: server umount lustre-MDT0001 complete [ 7468.788833] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7470.447955] Key type lgssc unregistered [ 7470.614989] LNet: 171374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7470.624395] LNetError: 171374:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7470.644950] LNet: Removed LNI 192.168.201.118@tcp [ 7471.108083] Key type .llcrypt unregistered [ 7471.109246] Key type ._llcrypt unregistered [ 7481.982591] Key type ._llcrypt registered [ 7481.983701] Key type .llcrypt registered [ 7482.048469] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7482.587461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7482.599913] alg: No test for adler32 (adler32-zlib) [ 7483.500952] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 7483.618916] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7485.239164] Key type lgssc registered [ 7485.751702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7490.394797] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7490.420336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7505.887676] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7511.007212] LustreError: 172575:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 7516.129252] LustreError: 172575:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7516.141607] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7516.152650] LustreError: MGC192.168.201.118@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7516.159855] LustreError: 172575:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7516.165988] LustreError: 172575:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 7516.172715] LustreError: 172575:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 7516.181601] LustreError: 172575:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 7516.227484] Lustre: server umount lustre-OST0000 complete [ 7516.231342] LustreError: 172575:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7521.463230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.721712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.062470] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7529.700033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7529.933772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7533.200858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7536.592901] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7538.796601] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7542.775568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7542.901301] LustreError: 174665:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7542.909100] Lustre: 174665:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.118@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7542.992090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7546.976905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7550.413982] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7553.505298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7553.509699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7553.515076] Lustre: Skipped 2 previous similar messages [ 7553.532658] Lustre: Skipped 2 previous similar messages [ 7554.529089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7554.538084] Lustre: Skipped 2 previous similar messages [ 7559.648574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7559.655793] Lustre: Skipped 2 previous similar messages [ 7564.772881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7564.778819] Lustre: Skipped 2 previous similar messages [ 7566.303798] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7566.361960] LustreError: 175317:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7566.366648] LustreError: 175317:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7566.419482] Lustre: server umount lustre-MDT0000 complete [ 7568.612259] LustreError: 173054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081767 with bad export cookie 9637904440772421461 [ 7568.615725] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7568.621030] LustreError: 173054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7568.887247] LustreError: 173788:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000004cae983 x1855016593809920/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7568.904896] LustreError: 175520:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7568.908242] LustreError: 175520:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7568.974197] Lustre: server umount lustre-MDT0001 complete [ 7580.218274] LustreError: 175721:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7580.220856] LustreError: 175721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7580.276816] Lustre: server umount lustre-OST0000 complete [ 7585.999854] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7591.824243] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7596.857196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7600.772655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7604.221283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7607.490971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7610.976165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7611.009586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7611.127736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7611.150499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7611.189694] Lustre: lustre-MDT0000: new disk, initializing [ 7611.230334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7611.242934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7613.221153] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7618.916427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7618.957350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7619.008682] Lustre: 178056:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7619.051077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7619.053523] Lustre: Skipped 1 previous similar message [ 7619.095119] Lustre: lustre-MDT0001: new disk, initializing [ 7619.140420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7619.161217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7619.169138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7621.453348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7626.198232] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7628.626958] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7632.207487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7632.239087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7632.365390] Lustre: lustre-OST0000: new disk, initializing [ 7632.368184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7632.420672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7635.154287] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7636.996784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7637.001741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7637.045742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7639.460536] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7642.308474] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7642.435607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7644.742323] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7644.870249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7647.209492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7647.213526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.219359] LustreError: Skipped 1 previous similar message [ 7647.220227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7647.239711] Lustre: Skipped 1 previous similar message [ 7651.960126] LustreError: 180994:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7651.963935] LustreError: 180994:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7652.034180] Lustre: server umount lustre-OST0000 complete [ 7654.884334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7654.895346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.449659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7657.452280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.453604] Lustre: Skipped 2 previous similar messages [ 7660.111934] LustreError: 181195:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7660.115190] LustreError: 181195:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7660.167196] Lustre: server umount lustre-MDT0000 complete [ 7662.203293] LustreError: 178048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081861 with bad export cookie 9637904440772422182 [ 7662.206853] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.214659] LustreError: 178048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7662.397275] Lustre: server umount lustre-MDT0001 complete [ 7666.634886] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:37:45 (1769081865) [ 7675.050101] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7676.456808] Key type lgssc unregistered [ 7676.634734] LNet: 182581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7676.637764] LNetError: 182581:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7676.647391] LNet: Removed LNI 192.168.201.118@tcp [ 7677.009331] Key type .llcrypt unregistered [ 7677.010572] Key type ._llcrypt unregistered [ 7686.403446] Key type ._llcrypt registered [ 7686.404705] Key type .llcrypt registered [ 7686.454829] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7686.959530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7686.969260] alg: No test for adler32 (adler32-zlib) [ 7687.861875] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 7687.962102] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7689.559166] Key type lgssc registered [ 7690.170462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7694.117643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7694.132282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.385172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.228796] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7701.459155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7701.636451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7703.386936] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7706.073435] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7708.103269] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7712.183243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7712.295760] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7712.297760] LustreError: 185394:0:(llog.c:1684:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7712.301726] Lustre: 185394:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.118@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7712.378424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7715.311852] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7718.070819] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7719.240076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.242326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7719.247699] Lustre: Skipped 1 previous similar message [ 7724.512336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7724.516010] Lustre: Skipped 4 previous similar messages [ 7725.392939] LustreError: 186046:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7725.437788] Lustre: server umount lustre-MDT0000 complete [ 7726.562993] LustreError: 183757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7726.570802] LustreError: 183757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7727.439067] LustreError: 185423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081926 with bad export cookie 16203572704408552404 [ 7727.439931] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7727.445101] LustreError: 185423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7728.607223] LustreError: 184470:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003d61f2a7 x1855016807714944/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7728.630693] LustreError: 186248:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7728.632757] LustreError: 186248:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7728.693979] Lustre: server umount lustre-MDT0001 complete [ 7740.990283] LustreError: 186449:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7740.994972] LustreError: 186449:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7741.052634] Lustre: server umount lustre-OST0000 complete [ 7746.223349] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7751.256463] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7755.813422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7759.238464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7762.775583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7765.806758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7768.716880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7768.745901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.857159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7768.876821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7768.916473] Lustre: lustre-MDT0000: new disk, initializing [ 7768.955562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.969723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7770.517488] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7775.352196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7775.384376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7775.422284] Lustre: 188782:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7775.445025] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7775.447242] Lustre: Skipped 1 previous similar message [ 7775.448954] LustreError: 189549:0:(llog.c:1684:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7775.452015] LustreError: 189549:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 7775.454642] Lustre: 189549:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7775.463928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7775.466895] Lustre: Skipped 1 previous similar message [ 7775.504176] Lustre: lustre-MDT0001: new disk, initializing [ 7775.542392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7775.559890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7775.567336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7777.200136] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7780.955732] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7782.549209] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7785.684483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7785.713219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7785.810990] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7785.812602] Lustre: Skipped 1 previous similar message [ 7785.814658] LustreError: 190506:0:(llog.c:1684:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7785.818719] LustreError: 190506:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 7785.821097] Lustre: 190506:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7785.825028] Lustre: 190506:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7785.834983] Lustre: lustre-OST0000: new disk, initializing [ 7785.837626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7785.877120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.429198] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7790.592101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7790.595894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7790.649285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7792.180972] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7793.908980] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7794.004246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7795.633908] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7795.723293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7800.804277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7800.807439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.812704] Lustre: Skipped 1 previous similar message [ 7800.815752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.817732] Lustre: Skipped 1 previous similar message [ 7802.548219] LustreError: 191724:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7802.550982] LustreError: 191724:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7802.589548] Lustre: server umount lustre-OST0000 complete [ 7805.924434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.927675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7805.934104] Lustre: Skipped 2 previous similar messages [ 7810.242774] LustreError: 191925:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7810.245703] LustreError: 191925:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7810.278950] Lustre: server umount lustre-MDT0000 complete [ 7811.041644] LustreError: 188794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7811.048171] LustreError: 188794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7811.865783] LustreError: 189564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082010 with bad export cookie 16203572704408553111 [ 7811.866812] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7811.870524] LustreError: 189564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7811.996554] Lustre: server umount lustre-MDT0001 complete [ 7815.804722] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:40:14 (1769082014) [ 7822.437061] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7826.554103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.726380] LustreError: 193493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7826.748224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.347858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7831.925028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7833.587558] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7834.694136] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7837.347688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7839.881694] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7843.631838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7843.665276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7843.753077] Lustre: lustre-OST0001: new disk, initializing [ 7843.755515] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7846.305710] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7848.951075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7848.957231] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7848.979300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7856.668878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7863.630208] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7869.436030] Lustre: DEBUG MARKER: ADD OST3 [ 7871.324644] loop: module loaded [ 7871.345192] loop0: detected capacity change from 0 to 204800000 [ 7871.422683] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7871.426544] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7871.430570] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7871.486355] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7873.815512] loop0: detected capacity change from 0 to 204800000 [ 7876.003749] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7876.032758] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7876.099112] Lustre: lustre-OST0003: new disk, initializing [ 7876.101565] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7876.153987] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7876.158366] Lustre: Skipped 3 previous similar messages [ 7876.648518] Lustre: DEBUG MARKER: STOP OST3 [ 7877.408810] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7877.457615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7877.463907] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7877.479450] Lustre: Failing over lustre-OST0003 [ 7877.485553] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7877.489988] LustreError: lustre-OST0003-osc-MDT0000: operation ost_create to node 0@lo failed: rc = -107 [ 7877.493534] LustreError: Skipped 1 previous similar message [ 7877.495353] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.501069] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 7877.503440] Lustre: Skipped 1 previous similar message [ 7877.513866] LustreError: 197802:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 7877.516885] LustreError: 197802:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7877.540061] Lustre: server umount lustre-OST0003 complete [ 7878.549495] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7879.327351] LustreError: 194948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7879.455570] LustreError: 197783:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7882.667166] Lustre: DEBUG MARKER: START OST3 again [ 7882.723886] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.337249] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7885.429570] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7887.259246] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7887.416578] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7887.416829] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7887.502932] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7901.734632] Lustre: Failing over lustre-OST0003 [ 7901.760107] LustreError: 199790:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 7901.762510] LustreError: 199790:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7901.779796] Lustre: server umount lustre-OST0003 complete [ 7902.690549] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.693628] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.700783] LustreError: 196273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7902.708385] LustreError: 196273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7911.395872] LustreError: 196267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.401977] LustreError: 196267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7916.521651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.527241] Lustre: Skipped 3 previous similar messages [ 7929.311110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7929.406580] Lustre: server umount lustre-MDT0000 complete [ 7930.876413] LustreError: 193474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082129 with bad export cookie 16203572704408554000 [ 7930.881147] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7930.881485] LustreError: 193474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7931.872165] LustreError: 193488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.872974] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.878166] LustreError: 193488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7931.883559] Lustre: Skipped 6 previous similar messages [ 7935.904477] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7935.907466] Lustre: Skipped 11 previous similar messages [ 7945.183112] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7945.223536] LustreError: 200264:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 7945.225819] LustreError: 200264:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7945.283564] Lustre: server umount lustre-MDT0001 complete [ 7946.974763] Lustre: server umount lustre-OST0000 complete [ 7948.542521] Lustre: server umount lustre-OST0001 complete [ 7951.281584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7954.882643] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7959.431771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7962.371067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7965.292963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7968.428952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7971.685795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7971.715720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7971.815986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7971.832067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7971.876410] Lustre: lustre-MDT0000: new disk, initializing [ 7971.922810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.927071] Lustre: Skipped 1 previous similar message [ 7971.935798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7973.587384] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7978.862605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7978.897987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7978.950454] Lustre: 202849:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7979.106220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7980.984273] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7985.115797] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7987.000645] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7989.896054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7989.922836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7990.047611] Lustre: lustre-OST0000: new disk, initializing [ 7990.052856] Lustre: Skipped 1 previous similar message [ 7990.056137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7990.061638] Lustre: Skipped 2 previous similar messages [ 7991.497618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7991.501074] Lustre: Skipped 1 previous similar message [ 7991.503587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7991.584835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7992.815391] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7996.656640] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7998.484483] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7998.572413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8000.316071] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8000.412987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8002.019264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.019768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8002.020793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.020798] Lustre: Skipped 2 previous similar messages [ 8002.024650] Lustre: Skipped 1 previous similar message [ 8002.026978] LustreError: Skipped 2 previous similar messages [ 8007.360995] Lustre: server umount lustre-OST0000 complete [ 8009.696958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8015.298186] LustreError: 205989:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8015.302564] LustreError: 205989:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 8015.335730] Lustre: server umount lustre-MDT0000 complete [ 8017.137711] LustreError: 203640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082216 with bad export cookie 16203572704408556282 [ 8017.140817] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8017.144422] LustreError: 203640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8017.284367] Lustre: server umount lustre-MDT0001 complete [ 8018.880487] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:43:37 (1769082217) [ 8026.790770] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8031.844356] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8036.178650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8039.176732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8042.020907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8045.347362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8048.628802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8048.655995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8048.784951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8048.804504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8048.848562] Lustre: lustre-MDT0000: new disk, initializing [ 8048.889298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8050.550968] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8055.880143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8055.919252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8055.974495] Lustre: 209030:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8056.141229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8058.284331] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8062.282759] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8064.120625] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8067.226723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8067.270085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8069.114909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8070.044465] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8073.907602] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8075.650394] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8075.741095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8077.488911] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8077.603433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8079.330406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8079.330563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8079.332383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8079.332389] Lustre: Skipped 6 previous similar messages [ 8079.339123] Lustre: Skipped 3 previous similar messages [ 8084.565302] Lustre: server umount lustre-OST0000 complete [ 8100.319151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8100.400956] Lustre: server umount lustre-MDT0000 complete [ 8102.283306] LustreError: 209023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082301 with bad export cookie 16203572704408557164 [ 8102.286692] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8102.288581] LustreError: 209023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8102.418599] Lustre: server umount lustre-MDT0001 complete [ 8106.218711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8106.476898] LustreError: 212845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.485437] LustreError: 212845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8106.516665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.519417] Lustre: Skipped 5 previous similar messages [ 8108.266539] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8112.124382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8114.093090] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8117.030845] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8118.899065] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8121.920241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8124.704835] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8126.920192] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8141.795385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.802718] LustreError: Skipped 2 previous similar messages [ 8145.202595] LustreError: 215209:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8145.204857] LustreError: 215209:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 8145.256184] Lustre: server umount lustre-OST0000 complete [ 8153.452987] Lustre: server umount lustre-MDT0000 complete [ 8155.249628] LustreError: 212826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082354 with bad export cookie 16203572704408558046 [ 8155.253162] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8155.254543] LustreError: 212826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8155.442932] Lustre: server umount lustre-MDT0001 complete [ 8263.040533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8264.367472] Key type lgssc unregistered [ 8264.534826] LNet: 216177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8264.538588] LNetError: 216177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8264.551086] LNet: Removed LNI 192.168.201.118@tcp [ 8264.915161] Key type .llcrypt unregistered [ 8264.918430] Key type ._llcrypt unregistered [ 8275.592896] Key type ._llcrypt registered [ 8275.594109] Key type .llcrypt registered [ 8275.648820] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8283.267598] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8283.732191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8283.750597] alg: No test for adler32 (adler32-zlib) [ 8284.627907] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 8284.730567] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8286.319318] Key type lgssc registered [ 8286.876083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8291.023847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8294.230968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8297.346753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8300.630317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8304.031875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8304.055535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8304.064536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.173631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8305.192241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8305.236405] Lustre: lustre-MDT0000: new disk, initializing [ 8305.270319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8305.282601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8307.192770] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8313.238784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8313.278506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8313.338420] Lustre: 219488:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8313.388477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8313.392960] Lustre: Skipped 1 previous similar message [ 8313.457621] Lustre: lustre-MDT0001: new disk, initializing [ 8313.503863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8313.527319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8313.536822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8315.555358] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8319.861266] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8321.722807] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8324.721793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8324.762666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8324.870485] Lustre: lustre-OST0000: new disk, initializing [ 8324.873831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8324.912879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8327.684114] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8331.936846] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8334.037216] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8334.341115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8334.345052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8334.397312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8335.154317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 8337.033538] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8337.126953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8339.425464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8339.425986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.429616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8339.429641] LustreError: Skipped 1 previous similar message [ 8339.432648] Lustre: Skipped 1 previous similar message [ 8340.021175] LustreError: 222434:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8340.077354] Lustre: server umount lustre-OST0000 complete [ 8344.545152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.545264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8344.545718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.545722] Lustre: Skipped 1 previous similar message [ 8344.553676] Lustre: Skipped 1 previous similar message [ 8349.664480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.667061] Lustre: Skipped 2 previous similar messages [ 8354.784907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.788468] Lustre: Skipped 1 previous similar message [ 8356.319331] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8356.361408] LustreError: 222635:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8356.364141] LustreError: 222635:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8356.400549] Lustre: server umount lustre-MDT0000 complete [ 8358.087472] LustreError: 219479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082557 with bad export cookie 17858267227006151968 [ 8358.090750] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8358.096917] LustreError: 219479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8358.159528] LustreError: 222837:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8358.161396] LustreError: 222837:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8358.226049] Lustre: server umount lustre-MDT0001 complete [ 8366.257358] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8371.088525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.288666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8373.002618] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8376.289700] LustreError: 224009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.299770] LustreError: 224009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8376.758431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8376.937077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8378.684362] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8380.019710] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8383.301562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8385.906230] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8389.968457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8390.014970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8390.097231] Lustre: lustre-OST0001: new disk, initializing [ 8390.100743] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8390.151977] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8390.159045] Lustre: Skipped 1 previous similar message [ 8392.914380] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8394.771362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8394.776200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8394.810802] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8399.593661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8403.987075] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8414.441644] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:50:12 (1769082612) [ 8417.251406] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.253024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8417.257655] Lustre: Skipped 2 previous similar messages [ 8417.259888] Lustre: Skipped 1 previous similar message [ 8417.282896] LustreError: 227385:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8417.285883] LustreError: 227385:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8417.317812] Lustre: server umount lustre-MDT0000 complete [ 8418.730564] LustreError: 223991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082617 with bad export cookie 17858267227006152829 [ 8418.735460] LustreError: 223991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8418.736183] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8418.853273] Lustre: server umount lustre-MDT0001 complete [ 8430.583894] LustreError: 227788:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8430.585761] LustreError: 227788:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8430.606219] Lustre: server umount lustre-OST0000 complete [ 8442.374224] LustreError: 227989:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8442.376631] LustreError: 227989:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8442.441854] Lustre: server umount lustre-OST0001 complete [ 8445.074826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8448.970202] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8453.049675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8455.600681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8458.060049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8461.044119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8464.293086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8464.320600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.425639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8464.440891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8464.479084] Lustre: lustre-MDT0000: new disk, initializing [ 8464.513217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8464.520897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8466.386143] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8471.571712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8471.602263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8471.647137] Lustre: 230172:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8471.764668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8473.657759] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8477.248981] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8478.916643] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8481.589093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8481.620553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8481.742094] Lustre: lustre-OST0000: new disk, initializing [ 8481.747601] Lustre: Skipped 1 previous similar message [ 8481.750262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8481.753101] Lustre: Skipped 2 previous similar messages [ 8483.395787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8483.399231] Lustre: Skipped 1 previous similar message [ 8483.400893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8483.440234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8484.070392] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8487.425367] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8489.044601] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8489.128596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8490.644753] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8490.729525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8493.545375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.547541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8493.549665] Lustre: Skipped 2 previous similar messages [ 8493.563636] Lustre: Skipped 1 previous similar message [ 8497.583093] LustreError: 233114:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8497.586747] LustreError: 233114:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8497.640967] Lustre: server umount lustre-OST0000 complete [ 8502.753578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8502.756192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.308672] Lustre: server umount lustre-MDT0000 complete [ 8506.702935] LustreError: 231920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082705 with bad export cookie 17858267227006154957 [ 8506.703726] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8506.707634] LustreError: 231920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8506.837131] Lustre: server umount lustre-MDT0001 complete [ 8510.047786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.226198] LustreError: 233991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8510.251503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8510.254597] Lustre: Skipped 2 previous similar messages [ 8511.706809] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8514.922862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8516.551796] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8518.698397] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8520.057473] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8522.535739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8524.868226] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8527.086099] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8674.273585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8674.275078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8674.279037] Lustre: Skipped 1 previous similar message [ 8674.283268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8674.287437] Lustre: Skipped 4 previous similar messages [ 8675.311401] LustreError: 236276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8675.314064] LustreError: 236276:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8675.357314] Lustre: server umount lustre-OST0000 complete [ 8678.879884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8678.884102] LustreError: Skipped 1 previous similar message [ 8691.168618] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8691.234120] Lustre: server umount lustre-MDT0000 complete [ 8692.535219] LustreError: 233972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082891 with bad export cookie 17858267227006155846 [ 8692.538175] LustreError: 233972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8692.538837] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8692.656890] Lustre: server umount lustre-MDT0001 complete [ 8790.847162] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8791.980611] Key type lgssc unregistered [ 8792.118384] LNet: 237250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8792.120704] LNetError: 237250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8792.129339] LNet: Removed LNI 192.168.201.118@tcp [ 8792.437094] Key type .llcrypt unregistered [ 8792.438295] Key type ._llcrypt unregistered [ 8801.856206] Key type ._llcrypt registered [ 8801.857261] Key type .llcrypt registered [ 8801.894361] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8812.557468] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8813.576489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8813.597289] alg: No test for adler32 (adler32-zlib) [ 8814.609737] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 8814.889037] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8816.575511] Key type lgssc registered [ 8817.461179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8824.049221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8828.894932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8833.088298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8836.653281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8844.948132] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8849.864098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8849.897989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8849.904652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.017057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8851.034636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8851.083180] Lustre: lustre-MDT0000: new disk, initializing [ 8851.118250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.128898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8853.123652] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8858.655166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8858.687957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8858.738909] Lustre: 241251:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8858.781240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8858.784066] Lustre: Skipped 1 previous similar message [ 8858.844173] Lustre: lustre-MDT0001: new disk, initializing [ 8858.911396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8858.945601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8858.954386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8860.956692] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8863.908923] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8867.105648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8867.139856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8867.260720] Lustre: lustre-OST0000: new disk, initializing [ 8867.263543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8867.310403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8867.474803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8867.479779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8867.531171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8869.730316] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8875.390228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8875.427412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8875.519790] Lustre: lustre-OST0001: new disk, initializing [ 8875.522134] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8875.576843] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8878.275958] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8882.184360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8882.190687] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8882.227175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8884.949621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8889.441527] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8900.064719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8900.070902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.082236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8900.617750] LustreError: 244773:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8900.657771] Lustre: server umount lustre-MDT0000 complete [ 8902.287813] LustreError: 241243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083101 with bad export cookie 4098390948142219681 [ 8902.293411] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8902.625482] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.626828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8902.629128] Lustre: Skipped 3 previous similar messages [ 8902.631357] Lustre: Skipped 1 previous similar message [ 8904.385749] LustreError: 244975:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8904.389038] LustreError: 244975:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8904.446323] Lustre: server umount lustre-MDT0001 complete [ 8908.153141] LustreError: 245176:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8908.162229] LustreError: 245176:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8908.190662] Lustre: server umount lustre-OST0000 complete [ 8909.830257] Lustre: server umount lustre-OST0001 complete [ 8915.537480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8919.279740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8919.645809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8922.804484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8926.413040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8926.809646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8934.697905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8938.576039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8938.584664] Lustre: lustre-MDT0000: reset Object Index mappings [ 8938.784066] LustreError: 248683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8938.797506] LustreError: 248683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8938.839738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.476176] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8944.097876] LustreError: 248683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.461984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8944.470196] Lustre: lustre-MDT0001: reset Object Index mappings [ 8946.662878] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8948.076967] Lustre: 249790:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8951.078579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8951.233983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8951.237144] Lustre: Skipped 1 previous similar message [ 8953.580842] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8954.339024] LustreError: 250148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.181509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8959.756617] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8963.178107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8963.178963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8968.968177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8970.644847] Lustre: 251641:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8978.914936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8978.916084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8978.919153] Lustre: Skipped 3 previous similar messages [ 8978.921583] Lustre: Skipped 2 previous similar messages [ 8983.173719] LustreError: 251744:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8983.175980] LustreError: 251744:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8983.204652] Lustre: server umount lustre-MDT0000 complete [ 8984.035555] LustreError: 248682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8984.042092] LustreError: 248682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8984.658406] LustreError: 251745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083183 with bad export cookie 4098390948142226457 [ 8984.660747] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8984.662659] LustreError: 251745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8984.784525] Lustre: server umount lustre-MDT0001 complete [ 8996.347293] LustreError: 252148:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8996.350888] LustreError: 252148:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8996.377778] Lustre: server umount lustre-OST0000 complete [ 9007.092867] LustreError: 252351:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9007.095732] LustreError: 252351:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9007.168705] Lustre: server umount lustre-OST0001 complete [ 9015.275535] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:00:13 (1769083213) [ 9021.413023] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9025.062509] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9028.739778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9031.305674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9033.773884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9036.618973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9039.338201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9039.372258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9039.464219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9039.476630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9039.519660] Lustre: lustre-MDT0000: new disk, initializing [ 9039.556966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9039.559991] Lustre: Skipped 1 previous similar message [ 9039.569977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9041.242051] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9049.103772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9049.164089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9049.219960] Lustre: 255869:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9049.765959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9049.768910] Lustre: Skipped 1 previous similar message [ 9049.850785] Lustre: lustre-MDT0001: new disk, initializing [ 9050.028431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9050.034363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9053.889515] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9060.318729] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9063.976969] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9069.629366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9069.680773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9069.825709] Lustre: lustre-OST0000: new disk, initializing [ 9069.828283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9071.720708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9071.727197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9071.792870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9073.177889] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9077.701500] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9079.912755] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9080.014956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9083.174975] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9083.402555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9087.458293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.465784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9087.466314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9087.470199] Lustre: Skipped 2 previous similar messages [ 9090.615774] LustreError: 258815:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9090.621162] LustreError: 258815:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9090.686530] Lustre: server umount lustre-OST0000 complete [ 9092.576715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9092.581208] Lustre: Skipped 1 previous similar message [ 9092.583668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9092.586323] Lustre: Skipped 1 previous similar message [ 9098.732561] Lustre: server umount lustre-MDT0000 complete [ 9100.635456] LustreError: 256659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083299 with bad export cookie 4098390948142228550 [ 9100.638595] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9100.639718] LustreError: 256659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9100.808263] Lustre: server umount lustre-MDT0001 complete [ 9109.174449] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9113.982655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9114.255797] LustreError: 260392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9114.286506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.291183] Lustre: Skipped 2 previous similar messages [ 9116.361920] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9120.661918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9123.166919] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9124.612924] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9128.276098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9131.366209] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9136.092428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9136.156618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9136.248878] Lustre: lustre-OST0001: new disk, initializing [ 9136.251758] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9138.150488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9138.155464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9138.215663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9139.081679] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9145.016021] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9146.622818] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9153.979609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9154.024882] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9157.088561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9157.094015] LustreError: Skipped 1 previous similar message [ 9157.098122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9157.109170] Lustre: Skipped 1 previous similar message [ 9157.111985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9157.115032] Lustre: Skipped 2 previous similar messages [ 9162.695250] LustreError: 263896:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9162.697669] LustreError: 263896:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9162.738150] Lustre: server umount lustre-MDT0000 complete [ 9164.346408] LustreError: 260370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083363 with bad export cookie 4098390948142229432 [ 9164.349588] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9164.351984] LustreError: 260370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9164.501931] Lustre: server umount lustre-MDT0001 complete [ 9176.468140] Lustre: server umount lustre-OST0000 complete [ 9188.934343] Lustre: server umount lustre-OST0001 complete [ 9196.921101] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9201.485027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9201.682727] LustreError: 265676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9201.692976] LustreError: 265676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9201.738374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.741034] Lustre: Skipped 3 previous similar messages [ 9203.493133] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9207.607057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9210.043172] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9211.655395] Lustre: 266783:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9214.437636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9217.375907] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9217.704730] LustreError: 267141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9217.723868] LustreError: 267141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9221.384216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9223.806120] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9226.541088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9226.547851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9232.725413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9234.325170] Lustre: 268639:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9241.691151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9241.709951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9245.935522] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:04:04 (1769083444) [ 9258.975990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9258.979211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9258.985219] Lustre: Skipped 3 previous similar messages [ 9258.987675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9258.990211] Lustre: Skipped 4 previous similar messages [ 9263.681953] LustreError: 269346:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9263.685112] LustreError: 269346:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 9263.721639] Lustre: server umount lustre-MDT0000 complete [ 9265.204818] LustreError: 265655:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083464 with bad export cookie 4098390948142235725 [ 9265.209102] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9265.211348] LustreError: 265655:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9267.679972] LustreError: 267162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9267.686034] LustreError: 267162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9271.446341] Lustre: server umount lustre-MDT0001 complete [ 9279.297907] Lustre: server umount lustre-OST0000 complete [ 9295.327159] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9295.391799] Lustre: server umount lustre-OST0001 complete [ 9297.867093] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9301.890470] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9305.973249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9308.835430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9311.633207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9314.283829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9320.609534] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9324.737773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9324.759428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9324.843537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9324.857768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9324.898273] Lustre: lustre-MDT0000: new disk, initializing [ 9324.938579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9326.421377] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9331.029630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9331.051993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9331.091635] Lustre: 272835:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9331.182225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9331.185098] Lustre: Skipped 4 previous similar messages [ 9331.197574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9332.651590] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9337.773309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9337.820628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9338.548636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9340.114464] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9344.982944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9345.028914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9346.748507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9347.423939] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9352.240732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9363.977172] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 07:06:02 (1769083562) [ 9367.009800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9367.013391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9367.018369] Lustre: Skipped 5 previous similar messages [ 9367.020607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9367.022816] Lustre: Skipped 8 previous similar messages [ 9370.977703] Lustre: server umount lustre-MDT0000 complete [ 9372.530927] LustreError: 272825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083571 with bad export cookie 4098390948142243593 [ 9372.533059] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9372.538840] LustreError: 272825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9372.641435] LustreError: 272841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9372.647659] LustreError: 272841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9372.731394] Lustre: server umount lustre-MDT0001 complete [ 9381.239509] Lustre: server umount lustre-OST0000 complete [ 9385.621581] Lustre: server umount lustre-OST0001 complete [ 9408.750534] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9422.055485] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9437.338483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9444.049257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9449.126443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9454.443695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9460.803934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9460.899465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.214577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9461.233592] Lustre: Skipped 2 previous similar messages [ 9461.252467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9461.255553] Lustre: Skipped 4 previous similar messages [ 9461.303659] Lustre: lustre-MDT0000: new disk, initializing [ 9461.309192] Lustre: Skipped 3 previous similar messages [ 9461.396683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9461.400762] Lustre: Skipped 3 previous similar messages [ 9465.304942] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9473.842106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9473.879550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9473.935382] Lustre: 279897:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9474.153173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9474.159801] Lustre: Skipped 2 previous similar messages [ 9477.493642] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9483.094177] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9485.781093] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9490.971722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9491.027486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9492.565621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9495.850695] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9502.275861] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9503.953689] Lustre: Mounted lustre-client [ 9504.731720] LustreError: 282402:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8c4012516800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9504.745533] LustreError: 282402:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9504.748915] LustreError: 282402:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 9504.798167] Lustre: Unmounted lustre-client [ 9512.929835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9512.929981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.936575] LustreError: Skipped 1 previous similar message [ 9512.940791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9512.944249] Lustre: Skipped 6 previous similar messages [ 9512.958778] Lustre: Skipped 6 previous similar messages [ 9517.882927] Lustre: server umount lustre-OST0000 complete [ 9527.029435] Lustre: server umount lustre-MDT0000 complete [ 9529.825741] LustreError: 279908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9529.837897] LustreError: 279908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9530.457602] LustreError: 281642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083729 with bad export cookie 4098390948142245119 [ 9530.467284] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9530.469224] LustreError: 281642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9530.722971] Lustre: server umount lustre-MDT0001 complete [ 9539.354547] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9541.511891] Key type lgssc unregistered [ 9559.775345] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp [ 9560.163141] LNet: 283679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9560.176600] LNetError: 283679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9634.983674] LNet: Removed LNI 192.168.201.118@tcp [ 9635.638554] Key type .llcrypt unregistered [ 9635.640738] Key type ._llcrypt unregistered [ 9648.890321] Key type ._llcrypt registered [ 9648.892742] Key type .llcrypt registered [ 9648.977367] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9660.337735] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9661.026590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9661.042509] alg: No test for adler32 (adler32-zlib) [ 9662.077687] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 9662.390073] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9664.071221] Key type lgssc registered [ 9664.895653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9671.265033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9675.978054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9681.246210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9687.039225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9688.818567] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:11:26 (1769083886) [ 9693.942168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9693.996816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9694.006069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9695.180509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9695.214668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9695.318867] Lustre: lustre-MDT0000: new disk, initializing [ 9695.403876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9695.421861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9699.304504] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9708.256811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9708.318831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9708.418877] Lustre: 286853:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9708.467447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9708.470546] Lustre: Skipped 1 previous similar message [ 9708.540953] Lustre: lustre-MDT0001: new disk, initializing [ 9708.617230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9708.646420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9708.652593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9711.701348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9717.260222] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9719.899917] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9725.234048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9725.296177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9725.467623] Lustre: lustre-OST0000: new disk, initializing [ 9725.471592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9725.545752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9729.914964] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9732.715612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9732.725328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9732.826370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9735.589721] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9743.332628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.336206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9743.346962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9743.355024] Lustre: Skipped 1 previous similar message [ 9748.451149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9748.459439] Lustre: Skipped 1 previous similar message [ 9748.858769] LustreError: 289295:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9749.058202] Lustre: server umount lustre-OST0000 complete [ 9753.572239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.584795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9753.590269] Lustre: Skipped 2 previous similar messages [ 9758.688157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9758.692714] Lustre: Skipped 2 previous similar messages [ 9763.809030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9763.822379] Lustre: Skipped 1 previous similar message [ 9766.368365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9766.436254] LustreError: 289496:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9766.440228] LustreError: 289496:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9766.536352] Lustre: server umount lustre-MDT0000 complete [ 9768.919906] LustreError: 286844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083967 with bad export cookie 4002602402557190357 [ 9768.921505] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9768.934422] LustreError: 286844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9769.045386] LustreError: 289699:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9769.050342] LustreError: 289699:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9769.170235] Lustre: server umount lustre-MDT0001 complete [ 9777.331354] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9779.384525] Key type lgssc unregistered [ 9779.642818] LNet: 290230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9779.648627] LNetError: 290230:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9780.709824] LNet: Removed LNI 192.168.201.118@tcp [ 9781.299318] Key type .llcrypt unregistered [ 9781.303609] Key type ._llcrypt unregistered [ 9850.414496] Key type ._llcrypt registered [ 9850.415881] Key type .llcrypt registered [ 9850.493731] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9850.691588] Key type .llcrypt unregistered [ 9850.693593] Key type ._llcrypt unregistered [ 9903.691202] Key type ._llcrypt registered [ 9903.692922] Key type .llcrypt registered [ 9903.790631] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9904.023466] Key type .llcrypt unregistered [ 9904.025720] Key type ._llcrypt unregistered [ 9925.646941] Key type ._llcrypt registered [ 9925.648788] Key type .llcrypt registered [ 9925.729993] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9925.961314] Key type .llcrypt unregistered [ 9925.962650] Key type ._llcrypt unregistered [ 9946.423808] Key type ._llcrypt registered [ 9946.426366] Key type .llcrypt registered [ 9946.493345] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9946.629416] Key type .llcrypt unregistered [ 9946.630394] Key type ._llcrypt unregistered [ 9996.703237] Key type ._llcrypt registered [ 9996.705297] Key type .llcrypt registered [ 9996.784152] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9996.944339] Key type .llcrypt unregistered [ 9996.945928] Key type ._llcrypt unregistered [10014.462332] Key type ._llcrypt registered [10014.466319] Key type .llcrypt registered [10014.556356] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10014.785359] Key type .llcrypt unregistered [10014.787796] Key type ._llcrypt unregistered [10033.652340] Key type ._llcrypt registered [10033.654369] Key type .llcrypt registered [10033.725429] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10033.889402] Key type .llcrypt unregistered [10033.890716] Key type ._llcrypt unregistered [10050.781903] Key type ._llcrypt registered [10050.783309] Key type .llcrypt registered [10050.839859] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10050.977724] Key type .llcrypt unregistered [10050.980885] Key type ._llcrypt unregistered [10068.213863] Key type ._llcrypt registered [10068.215231] Key type .llcrypt registered [10068.293604] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10068.446725] Key type .llcrypt unregistered [10068.447993] Key type ._llcrypt unregistered [10084.666073] Key type ._llcrypt registered [10084.668480] Key type .llcrypt registered [10084.736052] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10084.975127] Key type .llcrypt unregistered [10084.981532] Key type ._llcrypt unregistered [10119.050896] Key type ._llcrypt registered [10119.052632] Key type .llcrypt registered [10119.097708] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10119.234308] Key type .llcrypt unregistered [10119.236278] Key type ._llcrypt unregistered [10135.323321] Key type ._llcrypt registered [10135.324342] Key type .llcrypt registered [10135.373456] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10135.534807] Key type .llcrypt unregistered [10135.536200] Key type ._llcrypt unregistered [10153.632557] Key type ._llcrypt registered [10153.634319] Key type .llcrypt registered [10153.707585] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10153.885331] Key type .llcrypt unregistered [10153.887326] Key type ._llcrypt unregistered [10188.939868] Key type ._llcrypt registered [10188.941351] Key type .llcrypt registered [10188.986159] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10198.278814] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10198.910472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10198.927653] alg: No test for adler32 (adler32-zlib) [10199.880297] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [10200.017732] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10201.631214] Key type lgssc registered [10202.242669] Lustre: Echo OBD driver; http://www.lustre.org/ [10207.399566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10211.316507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10214.954107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10218.969684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10220.130544] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9885 sec ========== 07:20:18 (1769084418) [10221.288925] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:20:19 (1769084419) === [10223.355254] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:20:21 (1769084421) === [10245.633681] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10247.389150] Key type lgssc unregistered [10247.595623] LNet: 311974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10247.603183] LNetError: 311974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10247.626740] LNet: Removed LNI 192.168.201.118@tcp [10248.109577] Key type .llcrypt unregistered [10248.111237] Key type ._llcrypt unregistered