[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 433345151 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002256] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006559] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.010076] LSM: Security Framework initializing [ 0.011030] Yama: becoming mindful. [ 0.011785] SELinux: Initializing. [ 0.012042] *** VALIDATE selinux *** [ 0.020106] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025310] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027075] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028079] *** VALIDATE tmpfs *** [ 0.030164] *** VALIDATE proc *** [ 0.031216] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033197] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034121] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040073] debug: unmapping init [mem 0xffffffffaf059000-0xffffffffaf060fff] [ 0.042804] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043644] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044015] ... version: 2 [ 0.045009] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048007] ... max period: 00007fffffffffff [ 0.049008] ... fixed-purpose events: 3 [ 0.049903] ... event mask: 000000070000000f [ 0.051159] rcu: Hierarchical SRCU implementation. [ 0.053542] smp: Bringing up secondary CPUs ... [ 0.054418] x86: Booting SMP configuration: [ 0.055011] .... node #0, CPUs: #1 #2 #3 [ 0.057639] smp: Brought up 1 node, 4 CPUs [ 0.058934] smpboot: Max logical packages: 1 [ 0.059011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152696] node 0 deferred pages initialised in 92ms [ 0.155091] devtmpfs: initialized [ 0.156213] x86/mm: Memory block size: 128MB [ 0.158480] gcov: version magic: 0x41383552 [ 0.160695] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165252] pinctrl core: initialized pinctrl subsystem [ 0.167150] [ 0.167801] ************************************************************* [ 0.169007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171006] ** ** [ 0.172005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174007] ** ** [ 0.176008] ** This means that this kernel is built to expose internal ** [ 0.178011] ** IOMMU data structures, which may compromise security on ** [ 0.181010] ** your system. ** [ 0.183007] ** ** [ 0.185007] ** If you see this message and you are not debugging the ** [ 0.186006] ** kernel, report this immediately to your vendor! ** [ 0.188011] ** ** [ 0.190012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192009] ************************************************************* [ 0.195530] NET: Registered protocol family 16 [ 0.196348] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203094] cpuidle: using governor menu [ 0.204323] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205348] PCI: Using configuration type 1 for base access [ 0.207099] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.214010] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215089] cryptd: max_cpu_qlen set to 1000 [ 0.217242] ACPI: Added _OSI(Module Device) [ 0.218007] ACPI: Added _OSI(Processor Device) [ 0.218929] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225738] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.231343] ACPI: Interpreter enabled [ 0.233037] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234007] ACPI: Using IOAPIC for interrupt routing [ 0.235092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237241] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245950] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251038] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254969] acpiphp: Slot [2] registered [ 0.256058] acpiphp: Slot [5] registered [ 0.257056] acpiphp: Slot [6] registered [ 0.258035] acpiphp: Slot [7] registered [ 0.258944] acpiphp: Slot [8] registered [ 0.260065] acpiphp: Slot [9] registered [ 0.261018] acpiphp: Slot [10] registered [ 0.262054] acpiphp: Slot [3] registered [ 0.263048] acpiphp: Slot [4] registered [ 0.264048] acpiphp: Slot [11] registered [ 0.265048] acpiphp: Slot [12] registered [ 0.266060] acpiphp: Slot [13] registered [ 0.266965] acpiphp: Slot [14] registered [ 0.268047] acpiphp: Slot [15] registered [ 0.269071] acpiphp: Slot [16] registered [ 0.270067] acpiphp: Slot [17] registered [ 0.271069] acpiphp: Slot [18] registered [ 0.272049] acpiphp: Slot [19] registered [ 0.274060] acpiphp: Slot [20] registered [ 0.275057] acpiphp: Slot [21] registered [ 0.276084] acpiphp: Slot [22] registered [ 0.277049] acpiphp: Slot [23] registered [ 0.278067] acpiphp: Slot [24] registered [ 0.279058] acpiphp: Slot [25] registered [ 0.280086] acpiphp: Slot [26] registered [ 0.281063] acpiphp: Slot [27] registered [ 0.282081] acpiphp: Slot [28] registered [ 0.283048] acpiphp: Slot [29] registered [ 0.285103] acpiphp: Slot [30] registered [ 0.286049] acpiphp: Slot [31] registered [ 0.287040] PCI host bridge to bus 0000:00 [ 0.287966] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289010] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.292020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.294022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302749] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305913] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.313010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.317090] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.320011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.322008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.325436] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327506] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.329021] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.330574] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.336030] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357286] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.366012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.380011] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.387626] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.393008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.399013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.412012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.421677] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.428018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.433011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.446017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.454291] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.459011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.465010] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.479014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.488089] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.494014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.500013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.513014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.522199] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.528010] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.533012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.544013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.553073] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.554209] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.556293] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.558303] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.559132] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.562117] iommu: Default domain type: Passthrough [ 0.563306] SCSI subsystem initialized [ 0.565084] ACPI: bus type USB registered [ 0.566000] usbcore: registered new interface driver usbfs [ 0.566000] usbcore: registered new interface driver hub [ 0.569060] usbcore: registered new device driver usb [ 0.571158] pps_core: LinuxPPS API ver. 1 registered [ 0.573006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.576033] PTP clock support registered [ 0.578082] EDAC MC: Ver: 3.0.0 [ 0.579372] PCI: Using ACPI for IRQ routing [ 0.580602] NetLabel: Initializing [ 0.582006] NetLabel: domain hash size = 128 [ 0.583005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.584040] NetLabel: unlabeled traffic allowed by default [ 0.585070] vgaarb: loaded [ 0.586289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.588011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.595279] clocksource: Switched to clocksource kvm-clock [ 0.697624] VFS: Disk quotas dquot_6.6.0 [ 0.698787] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.700325] *** VALIDATE ramfs *** [ 0.701128] *** VALIDATE hugetlbfs *** [ 0.702082] pnp: PnP ACPI init [ 0.703714] pnp: PnP ACPI: found 6 devices [ 0.716940] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.718837] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.720093] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.722523] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.724796] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.726370] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.728775] NET: Registered protocol family 2 [ 0.731528] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.735072] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.738598] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.742339] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.744581] TCP: Hash tables configured (established 65536 bind 65536) [ 0.746358] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.749588] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.751554] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.754032] NET: Registered protocol family 1 [ 0.755746] RPC: Registered named UNIX socket transport module. [ 0.757549] RPC: Registered udp transport module. [ 0.758650] RPC: Registered tcp transport module. [ 0.759709] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.761659] NET: Registered protocol family 44 [ 0.763307] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.765701] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.767778] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.770599] PCI: CLS 0 bytes, default 64 [ 0.772048] Unpacking initramfs... [ 1.942550] debug: unmapping init [mem 0xffff88e6fcc54000-0xffff88e6fffbffff] [ 1.945649] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.947296] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 1.949113] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.360901] Initialise system trusted keyrings [ 2.362216] Key type blacklist registered [ 2.363485] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.369808] zbud: loaded [ 2.371890] *** VALIDATE nfs *** [ 2.372920] *** VALIDATE nfs4 *** [ 2.374173] pstore: using deflate compression [ 2.375886] Platform Keyring initialized [ 2.440558] NET: Registered protocol family 38 [ 2.441837] Key type asymmetric registered [ 2.442982] Asymmetric key parser 'x509' registered [ 2.444404] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.446210] io scheduler mq-deadline registered [ 2.447253] io scheduler kyber registered [ 2.448233] io scheduler bfq registered [ 2.449459] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.451418] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.453347] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.455113] ACPI: Power Button [PWRF] [ 2.458264] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.463973] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.471837] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.476094] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.485556] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.512155] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.538086] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.541221] Non-volatile memory driver v1.3 [ 2.542132] Linux agpgart interface v0.103 [ 2.565124] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.567412] vda: detected capacity change from 0 to 68624384 [ 2.578368] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.580575] vdb: detected capacity change from 0 to 1073741824 [ 2.590695] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.592728] vdc: detected capacity change from 0 to 2621440000 [ 2.603107] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.605072] vdd: detected capacity change from 0 to 2621440000 [ 2.616654] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.619681] vde: detected capacity change from 0 to 4294967296 [ 2.630727] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.632983] vdf: detected capacity change from 0 to 4294967296 [ 2.636856] libphy: Fixed MDIO Bus: probed [ 2.640372] usbcore: registered new interface driver usbserial_generic [ 2.641968] usbserial: USB Serial support registered for generic [ 2.643408] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.645981] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.647056] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.648449] mousedev: PS/2 mouse device common for all mice [ 2.650382] rtc_cmos 00:05: RTC can wake from S4 [ 2.651891] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.653996] rtc_cmos 00:05: registered as rtc0 [ 2.655239] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.657356] intel_pstate: CPU model not supported [ 2.659340] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.659586] hid: raw HID events driver (C) Jiri Kosina [ 2.663374] usbcore: registered new interface driver usbhid [ 2.664726] usbhid: USB HID core driver [ 2.665763] drop_monitor: Initializing network drop monitor service [ 2.665812] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.667538] Initializing XFRM netlink socket [ 2.670763] NET: Registered protocol family 10 [ 2.672546] Segment Routing with IPv6 [ 2.673482] NET: Registered protocol family 17 [ 2.674718] mpls_gso: MPLS GSO support [ 2.679321] RAS: Correctable Errors collector initialized. [ 2.680675] AVX version of gcm_enc/dec engaged. [ 2.681744] AES CTR mode by8 optimization enabled [ 2.728548] sched_clock: Marking stable (2728526914, 0)->(3491405456, -762878542) [ 2.730531] registered taskstats version 1 [ 2.731986] Loading compiled-in X.509 certificates [ 2.733158] zswap: loaded using pool lzo/zbud [ 2.748373] Key type big_key registered [ 2.756099] Key type encrypted registered [ 2.757206] ima: No TPM chip found, activating TPM-bypass! [ 2.758604] ima: Allocated hash algorithm: sha1 [ 2.759672] ima: No architecture policies found [ 2.760930] evm: Initialising EVM extended attributes: [ 2.761955] evm: security.selinux [ 2.762790] evm: security.ima [ 2.763566] evm: security.capability [ 2.764612] evm: HMAC attrs: 0x1 [ 2.766754] rtc_cmos 00:05: setting system clock to 2026-01-22 11:50:33 UTC (1769082633) [ 2.773185] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 2.776086] debug: unmapping init [mem 0xffffffffaed82000-0xffffffffaf058fff] [ 2.784190] Write protecting the kernel read-only data: 28672k [ 2.786425] debug: unmapping init [mem 0xffffffffad403000-0xffffffffad5fffff] [ 2.788226] debug: unmapping init [mem 0xffffffffadd14000-0xffffffffaddfffff] [ 2.811392] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.816154] systemd[1]: Detected virtualization kvm. [ 2.817473] systemd[1]: Detected architecture x86-64. [ 2.818657] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.843648] systemd[1]: No hostname configured. [ 2.845433] systemd[1]: Set hostname to . [ 2.847112] random: systemd: uninitialized urandom read (16 bytes read) [ 2.849469] systemd[1]: Initializing machine ID from random generator. [ 2.884218] random: ln: uninitialized urandom read (6 bytes read) [ 2.940437] random: systemd: uninitialized urandom read (16 bytes read) [ 2.942312] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 2.945084] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 2.948697] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.341919] device-mapper: uevent: version 1.0.3 [ 3.343192] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 3.789600] random: fast init done Starting dracut initqueue hook... [ 3.849478] virtio_net virtio0 ens2: renamed from eth0 [ 3.854457] scsi host0: ata_piix [ 3.874192] scsi host1: ata_piix [ 3.876300] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.879083] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.981581] dracut-initqueue[584]: RTNETLINK answers: File exists [ 8.269189] random: crng init done [ 8.270214] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 8.582799] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.455337] printk: systemd: 26 output lines suppressed due to ratelimiting [ 9.636378] SELinux: Disabled at runtime. [ 9.689230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.696531] systemd[1]: Detected virtualization kvm. [ 9.697703] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.048874] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.051360] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.055045] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.058189] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.061187] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.069711] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.073282] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ 10.095602] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 10.420330] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 10.672624] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.673441] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.779764] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.791620] EDAC sbridge: Ver: 1.1.2 [ 12.031119] Key type dns_resolver registered [ 12.288028] NFS: Registering the id_resolver key type [ 12.289636] Key type id_resolver registered [ 12.290432] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg303-server login: [ 33.497644] libcfs: loading out-of-tree module taints kernel. [ 33.514879] Key type ._llcrypt registered [ 33.515897] Key type .llcrypt registered [ 33.559470] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 40.522814] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 41.033824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 41.038856] alg: No test for adler32 (adler32-zlib) [ 41.983950] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 42.229027] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 43.824143] Key type lgssc registered [ 44.346115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 50.094291] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 50.621319] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 53.263531] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 55.459696] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 57.670826] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 62.303560] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 66.315055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.331407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 66.338327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 67.418736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 67.430326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 67.472158] Lustre: lustre-MDT0000: new disk, initializing [ 67.495833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 67.501504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 68.633911] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 73.415815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.436330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 73.460206] Lustre: 6473:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 73.469807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 73.471462] Lustre: Skipped 1 previous similar message [ 73.502402] Lustre: lustre-MDT0001: new disk, initializing [ 73.518401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 73.524697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 73.527534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 74.761488] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 76.935360] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 80.001826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.025144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 80.103285] Lustre: lustre-OST0000: new disk, initializing [ 80.105417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 80.120908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 80.403925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 80.406840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 80.428196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 81.885533] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 86.919925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.941641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 86.974136] Lustre: lustre-OST0001: new disk, initializing [ 86.976608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 86.991833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 88.302434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 88.305383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 88.329329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 88.751205] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.340118] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 97.780199] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 104.580519] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing check_logdir /tmp/testlogs/ [ 106.021688] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing yml_node [ 107.475173] Lustre: DEBUG MARKER: Client: 2.17.50.4 [ 108.328275] Lustre: DEBUG MARKER: MDS: 2.17.50.4 [ 109.175439] Lustre: DEBUG MARKER: OSS: 2.17.50.4 [ 109.707812] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 22 06:52:20 EST 2026 [ 115.626371] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 116.120955] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 129.504909] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 129.505239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 129.508418] Lustre: Skipped 3 previous similar messages [ 129.511017] Lustre: Skipped 2 previous similar messages [ 133.886522] LustreError: 12174:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 133.912697] Lustre: server umount lustre-MDT0000 complete [ 135.128589] LustreError: 7407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082765 with bad export cookie 8486739676748478414 [ 135.130564] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 135.134309] LustreError: 7407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 139.744671] LustreError: 10006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 139.744831] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 139.745060] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 139.750925] LustreError: 10006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 139.756377] Lustre: Skipped 1 previous similar message [ 140.768905] LustreError: 6481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 140.768946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 140.774955] LustreError: 6481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 140.776827] Lustre: Skipped 2 previous similar messages [ 141.249343] LustreError: 12377:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 141.252430] LustreError: 12377:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 141.310725] Lustre: server umount lustre-MDT0001 complete [ 148.718909] LustreError: 12578:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 148.721431] LustreError: 12578:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 148.739943] Lustre: server umount lustre-OST0000 complete [ 164.320119] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 164.342968] LustreError: 12779:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 164.345103] LustreError: 12779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 164.379502] Lustre: server umount lustre-OST0001 complete [ 166.444543] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 169.512816] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 172.926023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.069523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 177.151608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.356487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 181.670724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 181.692560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.780519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.793074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.832363] Lustre: lustre-MDT0000: new disk, initializing [ 181.858994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.867733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.247618] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 187.460487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.490374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.525810] Lustre: 14960:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.640381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 188.921408] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 191.986340] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 193.145481] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 195.407940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.430729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 195.526395] Lustre: lustre-OST0000: new disk, initializing [ 195.527671] Lustre: Skipped 1 previous similar message [ 195.530214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 195.533373] Lustre: Skipped 2 previous similar messages [ 195.561255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.563547] Lustre: Skipped 1 previous similar message [ 197.050130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.052567] Lustre: Skipped 1 previous similar message [ 197.054434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 197.090558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.427418] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 200.468714] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 201.880989] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 201.955694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 203.277454] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 203.352738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 206.063082] LustreError: 17895:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 206.065116] LustreError: 17895:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 206.101896] Lustre: server umount lustre-OST0000 complete [ 207.333112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.336122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 207.337832] Lustre: Skipped 1 previous similar message [ 212.449925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 212.451846] Lustre: Skipped 2 previous similar messages [ 217.572798] LustreError: 18095:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 217.574856] LustreError: 18095:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 217.597939] Lustre: server umount lustre-MDT0000 complete [ 218.776126] LustreError: 15741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082849 with bad export cookie 8486739676748479933 [ 218.778257] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 218.780743] LustreError: 15741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 228.384287] Lustre: server umount lustre-MDT0001 complete [ 230.230698] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:54:21 (1769082861) [ 236.213717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.343560] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 236.363108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.601083] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 240.157924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 241.441277] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 243.100239] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 244.155296] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 244.839429] Lustre: Failing over lustre-MDT0000 [ 244.843949] LustreError: 19464:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 244.846110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.847376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 244.847381] Lustre: Skipped 2 previous similar messages [ 244.847850] LustreError: 19464:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 244.850554] Lustre: Skipped 1 previous similar message [ 244.857253] LustreError: 19464:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 244.935505] LustreError: 20776:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 244.937528] LustreError: 20776:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 244.965277] Lustre: server umount lustre-MDT0000 complete [ 249.953085] LustreError: 20146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 249.957093] LustreError: 20146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 255.457493] LustreError: 20146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.462655] LustreError: 20146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 257.827795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.865373] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 259.203179] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 262.404472] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 263.138677] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 263.142300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 263.147535] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 265.506304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.507774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.512272] Lustre: Skipped 1 previous similar message [ 265.515857] Lustre: Skipped 2 previous similar messages [ 270.616836] Lustre: server umount lustre-MDT0000 complete [ 270.817164] LustreError: 19434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.823508] LustreError: 19434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 271.884349] LustreError: 19459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082902 with bad export cookie 8486739676748481340 [ 271.887927] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.888147] LustreError: 19459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 272.009375] Lustre: server umount lustre-MDT0001 complete [ 274.795729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.971096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.973678] Lustre: Skipped 2 previous similar messages [ 276.288610] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 279.213564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 280.722687] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 282.592908] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 283.785866] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 284.520794] Lustre: Failing over lustre-MDT0000 [ 284.600973] LustreError: 23772:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 284.602783] LustreError: 23772:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 284.632361] Lustre: server umount lustre-MDT0000 complete [ 284.640818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 284.643457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.761884] LustreError: 23142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 289.766907] LustreError: 23142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 297.690748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 297.730920] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.187645] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 302.524075] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 303.073547] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 303.076714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 303.079055] Lustre: Skipped 1 previous similar message [ 303.084078] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 305.505687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.509761] Lustre: Skipped 1 previous similar message [ 305.512964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.515413] Lustre: Skipped 1 previous similar message [ 310.614689] Lustre: server umount lustre-MDT0000 complete [ 311.862285] LustreError: 22409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082942 with bad export cookie 8486739676748482271 [ 311.865889] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.866381] LustreError: 22409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 311.994883] Lustre: server umount lustre-MDT0001 complete [ 314.941090] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:55:45 (1769082945) [ 320.275610] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 323.351690] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 326.731135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.761802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 330.738757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.832139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.972848] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 339.335202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.365848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.442023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 339.452425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 339.485299] Lustre: lustre-MDT0000: new disk, initializing [ 339.507856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.510765] Lustre: Skipped 2 previous similar messages [ 339.517801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 340.890846] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 345.392917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.410776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 345.435597] Lustre: 28415:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 345.527247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 347.014343] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 350.077790] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 351.278571] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 353.409272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 353.432381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 354.814268] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 354.841801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 354.847821] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 355.216597] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 359.022442] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 361.434904] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 365.026110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 365.028570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.032164] Lustre: Skipped 2 previous similar messages [ 370.147354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 370.149461] Lustre: Skipped 3 previous similar messages [ 371.008116] LustreError: 31052:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 371.011926] LustreError: 31052:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 371.064996] Lustre: server umount lustre-OST0000 complete [ 378.657489] Lustre: server umount lustre-MDT0000 complete [ 380.386587] LustreError: 29481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.408767] LustreError: 29481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 380.617896] LustreError: 28405:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083011 with bad export cookie 8486739676748482663 [ 380.620679] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.621882] LustreError: 28405:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 380.906119] Lustre: server umount lustre-MDT0001 complete [ 385.878805] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 387.079210] Key type lgssc unregistered [ 387.234748] LNet: 31987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.237982] LNetError: 31987:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 387.249447] LNet: Removed LNI 192.168.203.103@tcp [ 387.638129] Key type .llcrypt unregistered [ 387.639547] Key type ._llcrypt unregistered [ 398.608779] Key type ._llcrypt registered [ 398.609825] Key type .llcrypt registered [ 398.647016] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 398.981323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 399.009251] alg: No test for adler32 (adler32-zlib) [ 399.876539] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 399.956816] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 401.536142] Key type lgssc registered [ 401.886447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 404.558123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 404.565881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.717540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.051810] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 409.946230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 410.065952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 411.363971] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 413.109138] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 414.328933] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 416.527041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 416.634451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.442647] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 419.753340] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 420.239498] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 440.819329] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:57:51 (1769083071) [ 441.473834] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 442.102676] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 444.325605] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:57:55 (1769083075) [ 447.178057] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:57:58 (1769083078) [ 447.539577] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 450.494462] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 450.497221] Lustre: Skipped 2 previous similar messages [ 451.090208] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:58:01 (1769083081) [ 458.120992] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 458.124567] Lustre: Skipped 10 previous similar messages [ 458.972725] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:58:09 (1769083089) [ 459.875413] Lustre: *** cfs_fail_loc=131b, val=2*** [ 460.974452] Lustre: *** cfs_fail_loc=131b, val=2*** [ 463.598506] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:58:14 (1769083094) [ 467.643893] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 467.645935] Lustre: Skipped 7 previous similar messages [ 468.169711] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:58:19 (1769083099) [ 472.033946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.034270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 472.036149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.037892] Lustre: Skipped 1 previous similar message [ 477.153713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.155962] Lustre: Skipped 2 previous similar messages [ 482.273309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.275442] Lustre: Skipped 1 previous similar message [ 486.368159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.384650] LustreError: 39922:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 486.424102] Lustre: server umount lustre-OST0000 complete [ 492.000774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.003443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.008078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.759559] LustreError: 40127:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 493.761518] LustreError: 40127:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 493.789188] Lustre: server umount lustre-MDT0000 complete [ 495.024951] LustreError: 33338:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083125 with bad export cookie 13412051805421333483 [ 495.027527] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.028869] LustreError: 33338:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 495.085990] LustreError: 40328:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 495.087950] LustreError: 40328:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 495.134740] Lustre: server umount lustre-MDT0001 complete [ 498.837666] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 499.827360] Key type lgssc unregistered [ 499.962338] LNet: 40858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.964748] LNetError: 40858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 499.972277] LNet: Removed LNI 192.168.203.103@tcp [ 500.268905] Key type .llcrypt unregistered [ 500.270055] Key type ._llcrypt unregistered [ 506.124778] Key type ._llcrypt registered [ 506.125633] Key type .llcrypt registered [ 506.160952] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 512.484407] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 512.796804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 512.844434] alg: No test for adler32 (adler32-zlib) [ 513.701396] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 513.782292] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 515.360161] Key type lgssc registered [ 515.739263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 518.563759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 520.756361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 522.754554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 524.820976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 527.212592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 527.232744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 527.238924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.320401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 528.332189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 528.365648] Lustre: lustre-MDT0000: new disk, initializing [ 528.388636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.395393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 529.669927] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 533.730380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 533.752534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 533.784189] Lustre: 43958:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 533.805568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 533.807407] Lustre: Skipped 1 previous similar message [ 533.839537] Lustre: lustre-MDT0001: new disk, initializing [ 533.868471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 533.880052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 533.883552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 535.134472] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 538.230775] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 539.435879] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 541.667701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 541.690258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 541.780249] Lustre: lustre-OST0000: new disk, initializing [ 541.782384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 541.811055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.766837] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 546.856976] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 548.172706] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 550.390591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 550.394876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 550.420390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 551.287836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 552.631101] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 552.706312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 555.491512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 555.492102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.494776] LustreError: Skipped 1 previous similar message [ 555.500203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 555.501934] Lustre: Skipped 1 previous similar message [ 559.406470] LustreError: 46917:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 559.444262] Lustre: server umount lustre-OST0000 complete [ 564.704761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 564.706413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.708563] Lustre: Skipped 1 previous similar message [ 564.709808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 565.729755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 565.730068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.716922] LustreError: 47118:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 566.718918] LustreError: 47118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 566.744992] Lustre: server umount lustre-MDT0000 complete [ 567.948133] LustreError: 43949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083198 with bad export cookie 6519912146561699108 [ 567.951287] LustreError: 43949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 567.952018] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.014136] LustreError: 47319:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 568.015751] LustreError: 47319:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 568.063963] Lustre: server umount lustre-MDT0001 complete [ 571.139819] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:00:02 (1769083202) [ 573.449320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.655642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.976377] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 577.797212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 577.911349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 579.150174] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 580.953980] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 582.136381] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 584.457515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 586.470317] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 588.209597] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 605.873118] LustreError: 50254:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 606.222811] Lustre: Modifying parameter general.timeout=1 in log params [ 607.508464] Lustre: Modifying parameter general.timeout=5 in log params [ 607.511064] Lustre: Skipped 3 previous similar messages [ 609.516987] Lustre: Modifying parameter general.timeout=10 in log params [ 609.520574] Lustre: Skipped 4 previous similar messages [ 613.709976] Lustre: Modifying parameter general.timeout=18 in log params [ 613.713330] Lustre: Skipped 7 previous similar messages [ 622.155653] Lustre: Modifying parameter general.timeout=33 in log params [ 622.158269] Lustre: Skipped 14 previous similar messages [ 638.881599] Lustre: Modifying parameter general.timeout=54 in log params [ 638.892658] Lustre: Skipped 20 previous similar messages [ 671.219628] Lustre: Modifying parameter general.timeout=97 in log params [ 671.222368] Lustre: Skipped 42 previous similar messages [ 691.547194] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:02:02 (1769083322) [ 692.010022] hrtimer: interrupt took 2271169 ns [ 692.946712] Lustre: 57186:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 694.135746] Lustre: 57284:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 696.220412] Lustre: 57481:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 696.225639] Lustre: 57481:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 712.872425] Lustre: 59047:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 712.877663] Lustre: 59047:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 712.890927] LustreError: 59047:0:(mgs_handler.c:1129:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 715.150830] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:02:25 (1769083345) [ 722.913618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.920296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 722.927351] Lustre: Skipped 2 previous similar messages [ 722.935271] Lustre: Skipped 3 previous similar messages [ 728.034600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.038139] Lustre: Skipped 2 previous similar messages [ 728.404196] LustreError: 59744:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 728.407192] LustreError: 59744:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 728.456090] Lustre: server umount lustre-MDT0000 complete [ 730.421210] LustreError: 48018:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083361 with bad export cookie 6519912146561699976 [ 730.424712] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 730.426034] LustreError: 48018:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 730.598666] Lustre: server umount lustre-MDT0001 complete [ 742.774545] LustreError: 60146:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 742.777476] LustreError: 60146:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 742.840664] Lustre: server umount lustre-OST0000 complete [ 747.513355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 749.558324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 751.646854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 753.845599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 757.581247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.738974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 757.760505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 757.765942] Lustre: Skipped 55 previous similar messages [ 757.908063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.910098] Lustre: Skipped 1 previous similar message [ 759.892783] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 764.225672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 764.278385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 764.301594] Lustre: 61597:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 766.377183] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 768.994245] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 770.703759] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 774.698975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 774.871307] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 775.038814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.042889] Lustre: Skipped 1 previous similar message [ 778.332066] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 781.323485] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 782.227368] Lustre: 63833:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 788.965887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 788.967188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.970417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 788.979799] Lustre: Skipped 1 previous similar message [ 791.357081] LustreError: 64081:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 791.360375] LustreError: 64081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 791.413071] Lustre: server umount lustre-OST0000 complete [ 799.662610] Lustre: server umount lustre-MDT0000 complete [ 801.672916] LustreError: 63836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083432 with bad export cookie 6519912146561702573 [ 801.676610] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 801.677685] LustreError: 63836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 801.841510] Lustre: server umount lustre-MDT0001 complete [ 807.165315] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 808.692943] Key type lgssc unregistered [ 808.837514] LNet: 65023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.840432] LNetError: 65023:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.849650] LNet: Removed LNI 192.168.203.103@tcp [ 809.272305] Key type .llcrypt unregistered [ 809.273723] Key type ._llcrypt unregistered [ 821.172303] Key type ._llcrypt registered [ 821.173425] Key type .llcrypt registered [ 821.223306] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 821.592441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 821.599706] alg: No test for adler32 (adler32-zlib) [ 822.462252] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 822.545884] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 824.128294] Key type lgssc registered [ 824.573527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 828.151165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 828.160788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.430155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.021078] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 834.362172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 834.583240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 836.241631] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 838.480759] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 840.025135] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 842.787109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 843.079601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.859475] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 848.640992] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 862.975240] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 870.369693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 870.376584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.382578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.885719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.889869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.895109] Lustre: Skipped 1 previous similar message [ 870.901653] Lustre: Skipped 1 previous similar message [ 873.096732] LustreError: 69123:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 873.139917] Lustre: server umount lustre-MDT0000 complete [ 874.956200] LustreError: 66375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083505 with bad export cookie 3212318317243084413 [ 874.957420] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.960476] LustreError: 66375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 875.050726] LustreError: 69326:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 875.059677] LustreError: 69326:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 875.143436] Lustre: server umount lustre-MDT0001 complete [ 886.390052] LustreError: 69526:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 886.393283] LustreError: 69526:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 886.447762] Lustre: server umount lustre-OST0000 complete [ 890.529835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 892.357170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 894.083292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 896.144696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 899.425852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.567262] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 899.583303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 899.704515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.458809] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 905.255100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 905.297751] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 905.312510] Lustre: 70973:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 905.507772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.159547] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 909.537664] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 911.426594] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 914.752472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 914.853488] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 915.051990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.817081] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 920.298433] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.066724] Lustre: 73217:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 921.856832] Lustre: Modifying parameter general.jobid_var=disable in log params [ 925.667517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.668927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.671631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.676722] Lustre: Skipped 1 previous similar message [ 929.328075] LustreError: 73465:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 929.330863] LustreError: 73465:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 929.380073] Lustre: server umount lustre-OST0000 complete [ 930.785052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.787425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.790558] Lustre: Skipped 1 previous similar message [ 930.795306] Lustre: Skipped 2 previous similar messages [ 935.905306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.907367] Lustre: Skipped 1 previous similar message [ 945.120347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 945.189339] LustreError: 73672:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 945.193373] LustreError: 73672:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 945.237793] Lustre: server umount lustre-MDT0000 complete [ 946.147074] LustreError: 70984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.158473] LustreError: 70984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 948.203649] LustreError: 73219:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083578 with bad export cookie 3212318317243085232 [ 948.219225] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 948.470333] Lustre: server umount lustre-MDT0001 complete [ 958.493718] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 961.013191] Key type lgssc unregistered [ 961.227535] LNet: 74408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.232910] LNetError: 74408:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.278899] LNet: Removed LNI 192.168.203.103@tcp [ 963.059829] Key type .llcrypt unregistered [ 963.067916] Key type ._llcrypt unregistered [ 992.764711] Key type ._llcrypt registered [ 992.768875] Key type .llcrypt registered [ 992.918324] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 994.552492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 994.591097] alg: No test for adler32 (adler32-zlib) [ 995.915225] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 996.243321] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 997.989151] Key type lgssc registered [ 999.637281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.411595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1007.442471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.880605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.799041] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1017.340304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1017.740184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1020.297680] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1023.833417] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1026.356234] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1030.661560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.974751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.069841] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1038.663475] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1047.902613] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1048.548712] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1049.844560] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1049.848246] Lustre: Skipped 1 previous similar message [ 1052.240393] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1052.243090] Lustre: Skipped 3 previous similar messages [ 1056.702662] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1056.706956] Lustre: Skipped 6 previous similar messages [ 1065.315987] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1065.321170] Lustre: Skipped 12 previous similar messages [ 1081.481568] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1081.485272] Lustre: Skipped 24 previous similar messages [ 1118.808695] Lustre: 83558:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1119.596891] Lustre: Disabling parameter general.jobid_var= in log params [ 1119.599052] Lustre: Skipped 47 previous similar messages [ 1120.928696] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:09:11 (1769083751) [ 1122.218526] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1123.561412] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:09:13 (1769083753) [ 1126.369485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1126.372603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.386729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.425871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.426882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.278147] LustreError: 83854:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1132.361932] Lustre: server umount lustre-OST0000 complete [ 1135.588362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1135.595479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.604568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.608214] Lustre: Skipped 1 previous similar message [ 1138.658168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.658716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.669396] Lustre: Skipped 1 previous similar message [ 1140.745622] LustreError: 84057:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1140.748431] LustreError: 84057:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1140.801474] Lustre: server umount lustre-MDT0000 complete [ 1143.718863] LustreError: 75785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083774 with bad export cookie 14367092340471957236 [ 1143.724942] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.732218] LustreError: 75785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1143.866146] LustreError: 84258:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1143.869241] LustreError: 84258:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1144.008566] Lustre: server umount lustre-MDT0001 complete [ 1151.258987] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1152.953855] Key type lgssc unregistered [ 1153.157329] LNet: 84789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.165342] LNetError: 84789:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.181895] LNet: Removed LNI 192.168.203.103@tcp [ 1153.637105] Key type .llcrypt unregistered [ 1153.640470] Key type ._llcrypt unregistered [ 1155.780616] Key type ._llcrypt registered [ 1155.782584] Key type .llcrypt registered [ 1155.854112] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1157.753631] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules [ 1158.217523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.249030] alg: No test for adler32 (adler32-zlib) [ 1159.161454] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1159.277443] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1160.888153] Key type lgssc registered [ 1161.393621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1161.882158] LustreError: 85906:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1171.265508] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1176.652316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1183.184550] LustreError: 85906:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1183.207943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.595493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.694625] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1190.976129] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:10:21 (1769083821) [ 1193.160492] LustreError: 86944:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1193.164598] LustreError: 86944:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1193.167946] LustreError: 86944:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1193.237677] LustreError: 87716:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1193.311430] Lustre: server umount lustre-MDT0000 complete [ 1198.856570] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1200.195171] Key type lgssc unregistered [ 1200.330358] LNet: 88402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.336259] LNetError: 88402:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.349906] LNet: Removed LNI 192.168.203.103@tcp [ 1200.726035] Key type .llcrypt unregistered [ 1200.728270] Key type ._llcrypt unregistered [ 1208.566359] Key type ._llcrypt registered [ 1208.568390] Key type .llcrypt registered [ 1208.619529] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1209.116236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1209.130691] alg: No test for adler32 (adler32-zlib) [ 1210.033863] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1210.155597] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1211.760513] Key type lgssc registered [ 1212.306574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1215.941425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1215.949078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.159897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.812360] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1222.621757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1222.892839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1224.647535] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1226.995494] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1228.390852] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1231.008613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1231.301518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.870362] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1236.089946] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1270.135612] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:11:40 (1769083900) [ 1270.798918] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1271.481061] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:11:42 (1769083902) [ 1273.824988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1273.827527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.832656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1277.410113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1277.411562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.412412] Lustre: Skipped 1 previous similar message [ 1277.416122] Lustre: Skipped 1 previous similar message [ 1278.656181] LustreError: 92503:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1278.683630] Lustre: server umount lustre-MDT0000 complete [ 1280.132221] LustreError: 89557:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083910 with bad export cookie 9060705035991286418 [ 1280.136150] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.138601] LustreError: 89557:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1280.214403] LustreError: 92707:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1280.217332] LustreError: 92707:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1280.282760] Lustre: server umount lustre-MDT0001 complete [ 1292.018440] LustreError: 92908:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1292.021357] LustreError: 92908:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1292.140587] Lustre: server umount lustre-OST0000 complete [ 1296.223828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.380475] LustreError: 93537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.387651] LustreError: 93537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1296.447978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.861494] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1301.388902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1301.472952] LustreError: 93537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.720266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.259942] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1305.319729] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.682689] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1308.998322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1311.453509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1311.490344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1311.579971] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1311.584544] LustreError: 93524:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1311.597148] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1311.601364] LustreError: 95400:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1311.604555] LustreError: 95400:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1311.606413] LustreError: 95400:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1311.637709] Lustre: server umount lustre-OST0000 complete [ 1311.640378] LustreError: 95400:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1314.933212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1315.013072] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1315.017379] LustreError: 93524:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1315.030872] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1315.035088] LustreError: 95865:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1315.038184] LustreError: 95865:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1315.040747] LustreError: 95865:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1315.064166] Lustre: server umount lustre-OST0000 complete [ 1315.066244] LustreError: 95865:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1316.087129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1318.526068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1318.593032] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1318.595204] Lustre: Found index 0 for lustre-OST0000, updating log [ 1318.597849] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1318.620101] Lustre: lustre-OST0000: new disk, initializing [ 1318.622091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1318.713907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1320.701373] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1322.217630] Lustre: Failing over lustre-OST0000 [ 1322.244425] LustreError: 96947:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1322.246894] LustreError: 96947:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1322.287974] Lustre: server umount lustre-OST0000 complete [ 1326.818063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.819180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.823035] Lustre: Skipped 1 previous similar message [ 1326.825180] Lustre: Skipped 1 previous similar message [ 1332.195285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1332.198527] Lustre: Skipped 1 previous similar message [ 1336.290097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.292174] Lustre: Skipped 1 previous similar message [ 1338.336082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1338.376881] LustreError: 97148:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1338.379122] LustreError: 97148:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1338.403173] Lustre: server umount lustre-MDT0000 complete [ 1339.823483] LustreError: 93516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083970 with bad export cookie 9060705035991289148 [ 1339.826840] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.828174] LustreError: 93516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1339.945363] Lustre: server umount lustre-MDT0001 complete [ 1343.177659] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:12:53 (1769083973) [ 1349.172065] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1353.057038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.222901] LustreError: 98715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.256182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.754814] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1358.071911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1359.895632] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1360.936211] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1363.412781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1363.696328] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1365.843390] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1367.783117] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1367.789399] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 1367.797261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1369.146046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1369.172953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1369.202971] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1369.245333] Lustre: lustre-OST0001: new disk, initializing [ 1369.247765] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1369.357443] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 1369.364672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1371.073952] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 1371.089793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1371.093112] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1371.136830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1371.688597] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1375.160778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1376.228032] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 1376.234028] Lustre: Skipped 2 previous similar messages [ 1376.236331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1376.239986] LustreError: Skipped 2 previous similar messages [ 1378.288993] Lustre: lustre-OST0000: Recovery over after 0:11, of 1 clients 1 recovered and 0 were evicted. [ 1378.290826] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1378.293848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1381.024328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1387.936562] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 1388.000889] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1388.003678] LustreError: Skipped 1 previous similar message [ 1388.005635] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.599232] LustreError: 101954:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1393.601942] LustreError: 101954:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1393.625824] Lustre: server umount lustre-MDT0001 complete [ 1393.637423] LustreError: 99445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.642311] LustreError: 99445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1394.726995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1397.260916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1397.302519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1397.304877] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1397.323852] Lustre: 98704:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1397.621094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1397.623820] Lustre: Skipped 3 previous similar messages [ 1398.951582] LustreError: 102669:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1398.954860] LustreError: 102669:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1398.958123] LustreError: 98706:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1398.992668] LustreError: 102678:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.103@tcp: cfg command failed: rc = -17 [ 1398.996248] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1398.996248] [ 1399.328301] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1402.856967] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1403.132564] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 07:13:53 (1769084033) [ 1409.318513] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1415.791536] Lustre: 104561:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1424.477190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1425.725160] Lustre: 105692:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1479.650330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1479.650620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.656275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.659345] Lustre: Skipped 5 previous similar messages [ 1479.661721] Lustre: Skipped 7 previous similar messages [ 1485.193548] LustreError: 106128:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1485.195834] LustreError: 106128:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1485.224716] Lustre: server umount lustre-MDT0000 complete [ 1486.641192] LustreError: 99823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769084117 with bad export cookie 9060705035991290058 [ 1486.643219] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.646775] LustreError: 99823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1486.784803] Lustre: server umount lustre-MDT0001 complete [ 1499.148835] Lustre: server umount lustre-OST0000 complete [ 1510.812613] Lustre: server umount lustre-OST0001 complete [ 1512.839423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1515.889637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1516.258962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1519.469371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1523.012540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1523.446318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1531.007292] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1535.215476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.225258] Lustre: lustre-MDT0000: reset Object Index mappings [ 1535.468756] LustreError: 109605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.510979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.220491] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1540.731760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1540.741415] Lustre: lustre-MDT0001: reset Object Index mappings [ 1542.673212] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1543.790391] Lustre: 110719:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1546.285454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1547.618917] LustreError: 111076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.629510] LustreError: 111076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1547.632504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 1548.631730] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1550.697205] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1551.965414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1553.464809] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1553.486115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1554.685320] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1558.242237] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1559.716379] Lustre: 112572:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1585.719135] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:16:56 (1769084216) [ 1617.889040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1617.891911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.897034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1617.900152] Lustre: Skipped 4 previous similar messages [ 1620.106036] LustreError: 112925:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1620.108020] LustreError: 112925:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1620.148757] Lustre: server umount lustre-MDT0000 complete [ 1621.655783] LustreError: 109586:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769084252 with bad export cookie 9060705035991337014 [ 1621.660939] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1621.665222] LustreError: 109586:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1624.033083] LustreError: 112592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.038902] LustreError: 112592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1628.032391] Lustre: server umount lustre-MDT0001 complete [ 1635.666455] Lustre: server umount lustre-OST0000 complete [ 1643.368719] Lustre: server umount lustre-OST0001 complete [ 1645.755408] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 1649.366593] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1653.020141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1655.384661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1657.791192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1660.483028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1663.086252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1663.117563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.218127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1663.234336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1663.274218] Lustre: lustre-MDT0000: new disk, initializing [ 1663.314663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1664.760128] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1667.972808] Lustre: server umount lustre-MDT0000 complete [ 1668.983898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1671.598520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1671.625933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.740784] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1671.752359] Lustre: 116798:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1671.758041] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1671.840438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.843276] Lustre: Skipped 4 previous similar messages [ 1673.252377] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1675.003640] Lustre: server umount lustre-MDT0000 complete [ 1678.035618] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:18:28 (1769084308) [ 1678.627182] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1679.237950] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:18:30 (1769084310) [ 1685.461761] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1689.090408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.777269] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1693.650045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1693.673210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1693.707960] Lustre: 118811:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1693.731482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1693.733122] Lustre: Skipped 1 previous similar message [ 1693.770540] Lustre: lustre-MDT0001: new disk, initializing [ 1693.814313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1693.817591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1695.155133] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1697.549926] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1699.983566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1700.008657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1700.110742] Lustre: lustre-OST0000: new disk, initializing [ 1700.112688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1700.527849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1700.531745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1700.576171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1702.196523] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1706.445350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1706.476612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1708.169391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1708.174203] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1708.208921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1708.630925] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1713.332547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1714.612076] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1848.541536] Lustre: 127573:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1848.545743] Lustre: 127573:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1848.549049] Lustre: 127573:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1848.552224] Lustre: 127573:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1848.555466] Lustre: 127573:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1848.558353] Lustre: 127573:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1936.158981] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:22:46 (1769084566) [ 1950.176943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1950.179162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.181719] Lustre: Skipped 5 previous similar messages [ 1950.183267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.185009] Lustre: Skipped 8 previous similar messages [ 1953.862958] LustreError: 127780:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1953.865382] LustreError: 127780:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1953.927551] Lustre: server umount lustre-MDT0000 complete [ 1955.293408] LustreError: 127589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769084586 with bad export cookie 9060705035991393840 [ 1955.297461] LustreError: 127589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1955.297748] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1957.345249] LustreError: 121827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.352547] LustreError: 121827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1961.566323] Lustre: server umount lustre-MDT0001 complete [ 1977.312174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1977.353864] Lustre: server umount lustre-OST0000 complete [ 1984.731203] Lustre: server umount lustre-OST0001 complete [ 1986.844742] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 1990.093221] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1993.436378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1995.704752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1997.880193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2000.025680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2002.505807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2002.534121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.636609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2002.649295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2002.651474] Lustre: Skipped 1 previous similar message [ 2002.685843] Lustre: lustre-MDT0000: new disk, initializing [ 2002.687300] Lustre: Skipped 1 previous similar message [ 2002.713177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.715965] Lustre: Skipped 4 previous similar messages [ 2002.723103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2004.161390] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2008.463845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2008.489555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2008.525920] Lustre: 130567:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2008.636571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2009.953544] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2013.161596] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2014.387220] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2016.885434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2016.914572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2018.246801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2018.290862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2019.193028] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2022.366601] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2025.696731] Lustre: lustre-MDD0000: changelog on [ 2026.605029] Lustre: lustre-MDD0001: changelog on [ 2367.985100] Lustre: 130573:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2396.816473] Lustre: lustre-MDD0001: changelog off [ 2397.981595] Lustre: lustre-MDD0000: changelog off [ 2402.274437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.274554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2402.278259] Lustre: Skipped 5 previous similar messages [ 2402.281534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.284479] Lustre: Skipped 9 previous similar messages [ 2405.550079] LustreError: 139993:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2405.551942] LustreError: 139993:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2405.595568] Lustre: server umount lustre-OST0000 complete [ 2421.216082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2421.305593] Lustre: server umount lustre-MDT0000 complete [ 2422.498124] LustreError: 130558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085053 with bad export cookie 9060705035992193590 [ 2422.500640] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2422.503252] LustreError: 130558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2422.618428] Lustre: server umount lustre-MDT0001 complete [ 2426.054393] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2426.989499] Key type lgssc unregistered [ 2427.122300] LNet: 140928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2427.124800] LNetError: 140928:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2427.131269] LNet: Removed LNI 192.168.203.103@tcp [ 2427.404094] Key type .llcrypt unregistered [ 2427.405266] Key type ._llcrypt unregistered [ 2433.625685] Key type ._llcrypt registered [ 2433.626973] Key type .llcrypt registered [ 2433.662400] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2439.830826] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2440.228563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2440.235670] alg: No test for adler32 (adler32-zlib) [ 2441.077454] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2441.153640] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2442.728121] Key type lgssc registered [ 2443.070902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2445.999219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2448.209335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.320636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2452.498029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2454.835207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2454.849199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2454.854378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.926481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2455.936788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2455.970873] Lustre: lustre-MDT0000: new disk, initializing [ 2455.995247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.001132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2457.285996] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2461.266534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2461.286282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2461.314759] Lustre: 144084:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2461.335941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2461.337676] Lustre: Skipped 1 previous similar message [ 2461.370789] Lustre: lustre-MDT0001: new disk, initializing [ 2461.399334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2461.410052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2461.412529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2462.606707] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2465.611952] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2466.789727] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2469.056189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2469.076777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2469.161386] Lustre: lustre-OST0000: new disk, initializing [ 2469.163160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2469.190275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.017581] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2473.460988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2473.463940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2473.492244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2473.982906] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2476.115367] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_module obdecho/obdecho [ 2476.736753] LustreError: 146703:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2476.741374] LustreError: 146703:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 2477.326923] LustreError: 146799:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2482.144890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2482.147761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.153559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.682230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.683097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.688428] Lustre: Skipped 2 previous similar messages [ 2486.655723] LustreError: 147139:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2486.684339] Lustre: server umount lustre-MDT0000 complete [ 2487.921194] LustreError: 144075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085118 with bad export cookie 3969108628111919740 [ 2487.926106] LustreError: 144075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2487.926185] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.990561] LustreError: 147340:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2487.992347] LustreError: 147340:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2488.043397] Lustre: server umount lustre-MDT0001 complete [ 2499.506939] LustreError: 147540:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2499.509663] LustreError: 147540:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2499.556719] Lustre: server umount lustre-OST0000 complete [ 2506.265550] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2509.496127] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2512.818528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.010350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2517.079252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2519.237568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.552973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.580685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.662856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2521.673861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2521.707700] Lustre: lustre-MDT0000: new disk, initializing [ 2521.730168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.737129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2523.053412] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2527.210959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2527.231984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2527.269402] Lustre: 150482:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2527.292275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2527.294183] Lustre: Skipped 1 previous similar message [ 2527.326390] Lustre: lustre-MDT0001: new disk, initializing [ 2527.354451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2527.365598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2527.368684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2528.636804] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2531.754107] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2532.860453] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2535.056228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.082807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2536.250398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2536.254471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2536.282450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2537.024537] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2540.208188] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2541.666203] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2541.751520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2543.092037] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2543.176309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2546.659718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.660946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.662188] LustreError: Skipped 1 previous similar message [ 2546.667454] Lustre: Skipped 1 previous similar message [ 2546.669692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.672381] Lustre: Skipped 1 previous similar message [ 2549.933105] LustreError: 153420:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2549.935338] LustreError: 153420:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2549.976981] Lustre: server umount lustre-OST0000 complete [ 2551.777332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.777908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.782190] Lustre: Skipped 1 previous similar message [ 2556.897320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.899817] Lustre: Skipped 2 previous similar messages [ 2557.309711] LustreError: 153621:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2557.312239] LustreError: 153621:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2557.344727] Lustre: server umount lustre-MDT0000 complete [ 2558.597783] LustreError: 150474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085189 with bad export cookie 3969108628111920685 [ 2558.599320] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.603556] LustreError: 150474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2558.721071] Lustre: server umount lustre-MDT0001 complete [ 2559.943216] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 07:33:10 (1769085190) [ 2560.496647] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2561.069447] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 07:33:11 (1769085191) [ 2563.552578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.708354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.709992] Lustre: Skipped 1 previous similar message [ 2564.971898] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2567.868836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2569.343932] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2571.094884] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2572.248812] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2574.349397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2576.190015] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2577.823390] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2598.815047] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 2598.880958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2598.884219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.734891] LustreError: 156764:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2601.737882] LustreError: 156764:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2601.770936] Lustre: server umount lustre-MDT0000 complete [ 2603.038098] LustreError: 154477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085233 with bad export cookie 3969108628111921553 [ 2603.041632] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.041841] LustreError: 154477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2603.926661] LustreError: 154492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.933822] LustreError: 154492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2605.536799] LustreError: 154493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.544929] LustreError: 154493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.641491] LustreError: 154493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.646996] LustreError: 154493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2617.312118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2617.395320] Lustre: server umount lustre-MDT0001 complete [ 2619.523553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2623.597194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2623.671900] LustreError: 158027:0:(llog.c:1643:llog_backup()) MGC192.168.203.103@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2623.675200] Lustre: 158027:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.103@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2642.400868] LustreError: 158035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.405187] LustreError: 158035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2652.641073] LustreError: 158053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.646581] LustreError: 158053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2669.398225] LustreError: 158034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.403819] LustreError: 158034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 2684.896256] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2684.897691] Lustre: 158027:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.103@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2703.328965] LustreError: 158053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.335280] LustreError: 158053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2711.520227] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2711.522124] Lustre: 154488:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.103@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2725.417175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.420709] Lustre: Skipped 2 previous similar messages [ 2726.790982] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2727.459613] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2729.503750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.000273] LustreError: 154488:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 2747.877635] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x37151b449d8caad5 [ 2747.881884] Lustre: MGC192.168.203.103@tcp: Connection restored to 0@lo (at 0@lo) [ 2749.331846] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2751.137961] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2751.687111] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2752.903981] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2752.997903] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2756.538042] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2763.234881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2763.235326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.238610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.242110] Lustre: Skipped 4 previous similar messages [ 2763.245690] Lustre: Skipped 12 previous similar messages [ 2763.518126] LustreError: 159514:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2763.520076] LustreError: 159514:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2763.548741] Lustre: server umount lustre-MDT0000 complete [ 2764.802349] LustreError: 159515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085395 with bad export cookie 3969108628111928021 [ 2764.805598] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.807520] LustreError: 159515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2764.917894] Lustre: server umount lustre-MDT0001 complete [ 2775.389387] Lustre: server umount lustre-OST0000 complete [ 2782.370363] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2785.673025] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2788.886865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2790.860381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2792.818526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2794.834075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2797.010154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2797.029315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.101043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2797.112151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2797.114214] Lustre: Skipped 1 previous similar message [ 2797.151305] Lustre: lustre-MDT0000: new disk, initializing [ 2797.153267] Lustre: Skipped 1 previous similar message [ 2797.177377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.179418] Lustre: Skipped 1 previous similar message [ 2797.184687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2798.616437] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2802.845915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2802.867812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2802.900043] Lustre: 162855:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2802.986455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2804.313635] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2807.546881] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2808.787167] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2811.225554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2811.252868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2813.339465] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2815.480398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2815.513349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2816.455184] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2817.995689] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2818.081000] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2819.580753] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2819.673099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2820.577936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2820.579518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.581730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2820.586845] Lustre: Skipped 2 previous similar messages [ 2820.590891] Lustre: Skipped 1 previous similar message [ 2822.384116] LustreError: 165796:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2822.386139] LustreError: 165796:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2822.427786] Lustre: server umount lustre-OST0000 complete [ 2837.984123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2838.059944] Lustre: server umount lustre-MDT0000 complete [ 2839.408423] LustreError: 162848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085470 with bad export cookie 3969108628111944835 [ 2839.411113] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2839.414145] LustreError: 162848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2839.537431] Lustre: server umount lustre-MDT0001 complete [ 2842.645390] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 07:37:53 (1769085473) [ 2844.941874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2845.103142] LustreError: 166869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.108159] LustreError: 166869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 2846.452050] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2849.470597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2850.971807] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2852.778162] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2853.999868] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2856.309177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2858.451673] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2860.355134] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2874.729420] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 07:38:25 (1769085505) [ 2879.971880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2879.974339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.975913] LustreError: Skipped 2 previous similar messages [ 2879.978943] Lustre: Skipped 2 previous similar messages [ 2881.814714] Lustre: server umount lustre-OST0000 complete [ 2885.090771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.092763] Lustre: Skipped 9 previous similar messages [ 2897.376133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2897.413534] LustreError: 169536:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2897.415325] LustreError: 169536:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 2897.442601] Lustre: server umount lustre-MDT0000 complete [ 2898.759568] LustreError: 166849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085529 with bad export cookie 3969108628111945703 [ 2898.764103] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.764913] LustreError: 166849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2898.879509] Lustre: server umount lustre-MDT0001 complete [ 2902.620217] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2903.702914] Key type lgssc unregistered [ 2903.846348] LNet: 170270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2903.848941] LNetError: 170270:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2903.856326] LNet: Removed LNI 192.168.203.103@tcp [ 2904.214110] Key type .llcrypt unregistered [ 2904.215423] Key type ._llcrypt unregistered [ 2911.974034] Key type ._llcrypt registered [ 2911.975138] Key type .llcrypt registered [ 2912.012109] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2912.430197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2912.460128] alg: No test for adler32 (adler32-zlib) [ 2913.325372] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2913.426714] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2915.008107] Key type lgssc registered [ 2915.376149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2918.102924] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2918.110810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2933.536395] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2938.656216] LustreError: 171469:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 2943.776458] LustreError: 171469:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2943.781048] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2943.784320] LustreError: MGC192.168.203.103@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2943.788845] LustreError: 171469:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2943.792481] LustreError: 171469:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 2943.795125] LustreError: 171469:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2943.797237] LustreError: 171469:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2943.823732] Lustre: server umount lustre-OST0000 complete [ 2943.825242] LustreError: 171469:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2946.838061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.995537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.374445] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2952.317819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2952.450919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2953.733484] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2955.555213] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2956.800502] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2959.050287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2959.115687] LustreError: 173561:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2959.119465] Lustre: 173561:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.203.103@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2959.161272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.026374] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2962.765121] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2968.032949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.035262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.039152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2968.546789] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.547753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2968.550216] Lustre: Skipped 1 previous similar message [ 2968.553524] Lustre: Skipped 1 previous similar message [ 2969.598520] LustreError: 174212:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2969.600480] LustreError: 174212:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2969.626522] Lustre: server umount lustre-MDT0000 complete [ 2970.864373] LustreError: 174214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085601 with bad export cookie 795039877099133818 [ 2970.867463] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2970.868704] LustreError: 174214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2970.934909] LustreError: 174415:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2970.936872] LustreError: 174415:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2970.979896] Lustre: server umount lustre-MDT0001 complete [ 2981.430080] LustreError: 174615:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2981.433502] LustreError: 174615:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2981.479374] Lustre: server umount lustre-OST0000 complete [ 2984.538091] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2988.083772] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2991.671113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.830072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2996.005837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2998.136900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3000.439465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3000.460365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.548923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3000.561776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3000.607945] Lustre: lustre-MDT0000: new disk, initializing [ 3000.633828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.640957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3001.909203] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3006.029315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3006.051271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3006.080786] Lustre: 176947:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3006.103322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3006.105605] Lustre: Skipped 1 previous similar message [ 3006.146819] Lustre: lustre-MDT0001: new disk, initializing [ 3006.181350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3006.195506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3006.198947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3007.480580] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3010.580581] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3011.834450] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3014.086666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.108940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3014.194359] Lustre: lustre-OST0000: new disk, initializing [ 3014.196312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3014.225546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.930516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3015.933839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3015.975943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3016.206970] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3019.379951] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3020.734548] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3020.806585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3022.159391] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3022.228624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3026.401568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3026.402672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.404528] LustreError: Skipped 1 previous similar message [ 3026.405208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3026.410292] Lustre: Skipped 1 previous similar message [ 3031.523621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.526059] Lustre: Skipped 2 previous similar messages [ 3036.645741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.648874] Lustre: Skipped 1 previous similar message [ 3037.152234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3037.182146] LustreError: 179884:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3037.186335] LustreError: 179884:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3037.238488] Lustre: server umount lustre-OST0000 complete [ 3041.761874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.768131] Lustre: Skipped 1 previous similar message [ 3044.824874] Lustre: server umount lustre-MDT0000 complete [ 3046.033506] LustreError: 176937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085676 with bad export cookie 795039877099134518 [ 3046.036021] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3046.037324] LustreError: 176937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3046.110168] LustreError: 180290:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3046.112011] LustreError: 180290:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3046.174561] Lustre: server umount lustre-MDT0001 complete [ 3049.122939] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 07:41:19 (1769085679) [ 3054.962493] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3055.979818] Key type lgssc unregistered [ 3056.106349] LNet: 181471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3056.108740] LNetError: 181471:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3056.116282] LNet: Removed LNI 192.168.203.103@tcp [ 3056.409408] Key type .llcrypt unregistered [ 3056.410896] Key type ._llcrypt unregistered [ 3063.243239] Key type ._llcrypt registered [ 3063.244323] Key type .llcrypt registered [ 3063.279550] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3063.650042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3063.681617] alg: No test for adler32 (adler32-zlib) [ 3064.521992] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3064.599051] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3066.176245] Key type lgssc registered [ 3066.541518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3069.106078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3069.111851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.269287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.522188] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3074.287524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3074.408835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3075.760609] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3077.744584] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3078.924556] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3081.558969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3081.628574] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3081.629869] LustreError: 184286:0:(llog.c:1684:llog_backup()) MGC192.168.203.103@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3081.633249] Lustre: 184286:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.203.103@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3081.670060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.612093] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3085.616951] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3086.377197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.380321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3086.380497] Lustre: Skipped 2 previous similar messages [ 3086.382940] Lustre: Skipped 1 previous similar message [ 3091.427150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.430772] Lustre: Skipped 2 previous similar messages [ 3092.476168] LustreError: 184934:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3092.499073] Lustre: server umount lustre-MDT0000 complete [ 3093.886084] LustreError: 184315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085724 with bad export cookie 2445042679802174045 [ 3093.892239] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3093.894852] LustreError: 184315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3093.973071] LustreError: 185139:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3093.975251] LustreError: 185139:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3094.021893] Lustre: server umount lustre-MDT0001 complete [ 3105.713759] LustreError: 185340:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3105.715737] LustreError: 185340:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3105.754254] Lustre: server umount lustre-OST0000 complete [ 3109.253650] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3113.703526] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3117.357510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3119.708698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3121.911642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3124.102571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3126.440667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3126.465711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.562452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3126.574541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3126.617178] Lustre: lustre-MDT0000: new disk, initializing [ 3126.644936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.651569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3127.998336] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3132.164140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.192142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3132.226257] Lustre: 187672:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3132.251055] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3132.252421] Lustre: Skipped 1 previous similar message [ 3132.254044] LustreError: 188438:0:(llog.c:1684:llog_backup()) MGC192.168.203.103@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3132.256976] LustreError: 188438:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3132.258973] Lustre: 188438:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.103@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3132.268824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3132.271604] Lustre: Skipped 1 previous similar message [ 3132.308725] Lustre: lustre-MDT0001: new disk, initializing [ 3132.342334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3132.355768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3132.359213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3133.646814] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3136.760250] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3138.023407] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3140.227197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.253155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3140.342819] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3140.344237] Lustre: Skipped 1 previous similar message [ 3140.345534] LustreError: 189395:0:(llog.c:1684:llog_backup()) MGC192.168.203.103@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3140.348920] LustreError: 189395:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3140.350948] Lustre: 189395:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.103@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3140.353875] Lustre: 189395:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3140.362116] Lustre: lustre-OST0000: new disk, initializing [ 3140.364441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3140.392175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3142.214142] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3142.719801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3142.723886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3142.760908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3145.348654] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3146.704661] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3146.781265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3148.080119] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3148.146526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3150.831172] LustreError: 190611:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3150.833149] LustreError: 190611:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3150.872939] Lustre: server umount lustre-OST0000 complete [ 3152.866733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3152.867070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.868052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.874378] Lustre: Skipped 1 previous similar message [ 3157.985260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3157.987297] Lustre: Skipped 2 previous similar messages [ 3162.083912] LustreError: 190811:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3162.085706] LustreError: 190811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3162.113136] Lustre: server umount lustre-MDT0000 complete [ 3163.105105] LustreError: 187683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.109090] LustreError: 187683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3163.275182] LustreError: 187664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085794 with bad export cookie 2445042679802174731 [ 3163.278605] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.280519] LustreError: 187664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3173.859732] LustreError: 191014:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3173.861610] LustreError: 191014:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3173.900649] Lustre: server umount lustre-MDT0001 complete [ 3176.826408] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 07:43:27 (1769085807) [ 3181.938376] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3185.335643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.482174] LustreError: 192381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.495457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.829554] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3189.707537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3191.053135] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3191.978895] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3194.167948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3196.236610] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3199.100544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.126605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3199.197265] Lustre: lustre-OST0001: new disk, initializing [ 3199.199702] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3201.235115] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3206.664859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3206.668698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3206.719301] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3206.850958] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3210.573264] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3216.269598] Lustre: DEBUG MARKER: ADD OST3 [ 3217.965368] loop: module loaded [ 3217.987848] loop0: detected capacity change from 0 to 204800000 [ 3218.061517] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3218.064909] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3218.068419] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3218.116492] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3220.291185] loop0: detected capacity change from 0 to 204800000 [ 3222.276501] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3222.302633] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3222.371714] Lustre: lustre-OST0003: new disk, initializing [ 3222.373499] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3222.412024] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3222.414100] Lustre: Skipped 3 previous similar messages [ 3223.029935] Lustre: DEBUG MARKER: STOP OST3 [ 3223.695062] Lustre: Failing over lustre-OST0003 [ 3223.725755] LustreError: 196679:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3223.729137] LustreError: 196679:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3223.747216] Lustre: server umount lustre-OST0003 complete [ 3224.265699] LustreError: 194817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.577994] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3228.343231] Lustre: DEBUG MARKER: START OST3 again [ 3229.089775] LustreError: 193834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.093763] LustreError: 193834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3230.530954] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3230.609179] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3232.097858] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3232.104793] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3232.305338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3232.308288] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3232.324773] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3232.457866] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3245.991574] Lustre: Failing over lustre-OST0003 [ 3246.033606] Lustre: server umount lustre-OST0003 complete [ 3246.064027] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.065317] LustreError: 193832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3246.068107] Lustre: Skipped 1 previous similar message [ 3246.075606] LustreError: 193832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3256.292725] LustreError: 193833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.298157] LustreError: 193833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3261.414947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.418307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3261.418877] Lustre: Skipped 1 previous similar message [ 3261.419456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3264.918000] LustreError: 198938:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 3264.920338] LustreError: 198938:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3264.962922] Lustre: server umount lustre-MDT0000 complete [ 3266.643571] LustreError: 192363:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085897 with bad export cookie 2445042679802175599 [ 3266.649644] LustreError: 192363:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3266.653866] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3271.652254] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.659580] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3271.662617] Lustre: Skipped 4 previous similar messages [ 3272.891977] Lustre: server umount lustre-MDT0001 complete [ 3274.376453] Lustre: server umount lustre-OST0000 complete [ 3275.691865] Lustre: server umount lustre-OST0001 complete [ 3277.955781] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3281.138809] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3284.899873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3287.119797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3289.241994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3291.483330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3293.779196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3293.802985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.906274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3293.918485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3293.960488] Lustre: lustre-MDT0000: new disk, initializing [ 3293.987110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.990666] Lustre: Skipped 1 previous similar message [ 3293.998378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3295.448123] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3299.777723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3299.802754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3299.838631] Lustre: 201722:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3299.954269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3301.251935] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3304.436392] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3305.704776] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3308.108107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3308.150125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3310.010075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3310.012479] Lustre: Skipped 1 previous similar message [ 3310.014080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3310.050237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3310.234621] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3313.412149] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3314.972512] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3315.038117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3316.290108] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3316.378747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3320.291673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3320.291982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.293706] LustreError: Skipped 1 previous similar message [ 3320.299235] Lustre: Skipped 1 previous similar message [ 3320.301365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3320.304508] Lustre: Skipped 1 previous similar message [ 3323.160840] Lustre: server umount lustre-OST0000 complete [ 3325.410423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3330.694115] LustreError: 204863:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3330.697734] LustreError: 204863:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3330.735872] Lustre: server umount lustre-MDT0000 complete [ 3332.165832] LustreError: 203467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085962 with bad export cookie 2445042679802177867 [ 3332.169548] LustreError: 203467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3332.169883] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3332.300957] Lustre: server umount lustre-MDT0001 complete [ 3333.706258] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:46:04 (1769085964) [ 3340.395818] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3343.840968] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3347.374572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3350.142863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.703521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3355.374855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.085477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.108854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.197404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3358.210097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3358.212653] Lustre: Skipped 3 previous similar messages [ 3358.257662] Lustre: lustre-MDT0000: new disk, initializing [ 3358.259215] Lustre: Skipped 2 previous similar messages [ 3358.298391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3359.599745] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3364.293098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3364.325192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3364.374814] Lustre: 207905:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3364.503225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3366.058280] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3369.687056] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3371.281931] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3373.647885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3373.676140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3375.590408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3375.899871] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3379.256242] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3380.556119] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3380.619968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3382.152484] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3382.261286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3385.827694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3385.830068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.832958] LustreError: Skipped 1 previous similar message [ 3385.840556] Lustre: Skipped 2 previous similar messages [ 3385.841972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3385.850685] Lustre: Skipped 5 previous similar messages [ 3389.221902] Lustre: server umount lustre-OST0000 complete [ 3395.552763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3397.209355] Lustre: server umount lustre-MDT0000 complete [ 3398.736939] LustreError: 207897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086029 with bad export cookie 2445042679802178749 [ 3398.743569] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.745490] LustreError: 207897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3398.901807] Lustre: server umount lustre-MDT0001 complete [ 3402.179845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.331176] LustreError: 211721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.336303] LustreError: 211721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3403.620513] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3406.929545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3408.647090] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3410.746628] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3412.389173] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3415.580314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3417.863898] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3420.294400] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3439.587367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3439.590324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.601128] Lustre: Skipped 3 previous similar messages [ 3443.856467] Lustre: server umount lustre-OST0000 complete [ 3451.450442] Lustre: server umount lustre-MDT0000 complete [ 3452.987661] LustreError: 211701:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086083 with bad export cookie 2445042679802179624 [ 3452.991804] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.993320] LustreError: 211701:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3453.125534] Lustre: server umount lustre-MDT0001 complete [ 3552.436961] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3553.762910] Key type lgssc unregistered [ 3553.924695] LNet: 215055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3553.930220] LNetError: 215055:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3553.941604] LNet: Removed LNI 192.168.203.103@tcp [ 3554.431195] Key type .llcrypt unregistered [ 3554.433249] Key type ._llcrypt unregistered [ 3563.969462] Key type ._llcrypt registered [ 3563.970580] Key type .llcrypt registered [ 3564.007428] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3570.861397] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3571.278901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3571.288921] alg: No test for adler32 (adler32-zlib) [ 3572.126526] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3572.203279] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3573.776190] Key type lgssc registered [ 3574.325845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3577.656482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3580.298049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3582.952885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3585.180875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3588.167808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3588.197786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3588.206963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.313570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3589.331304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3589.384027] Lustre: lustre-MDT0000: new disk, initializing [ 3589.449661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.461355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3591.022835] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3596.123684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3596.163388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3596.217183] Lustre: 218372:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3596.269548] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3596.272394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3596.275657] Lustre: Skipped 1 previous similar message [ 3596.317984] Lustre: lustre-MDT0001: new disk, initializing [ 3596.358203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3596.372077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3596.375640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3597.981375] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3601.671539] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3602.898331] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3605.624062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3605.645640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3605.737311] Lustre: lustre-OST0000: new disk, initializing [ 3605.740220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3605.765197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3607.957306] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3609.858286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3609.862116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3609.911189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3611.284456] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3612.852741] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3612.950893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3614.403605] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3614.485690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3620.324686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3620.325553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.327110] LustreError: Skipped 1 previous similar message [ 3620.328089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3620.332122] Lustre: Skipped 1 previous similar message [ 3621.423118] LustreError: 221308:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3621.459445] Lustre: server umount lustre-OST0000 complete [ 3625.441833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.442499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.447427] Lustre: Skipped 1 previous similar message [ 3625.449122] Lustre: Skipped 1 previous similar message [ 3630.562107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.567276] Lustre: Skipped 2 previous similar messages [ 3635.681202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3635.682885] Lustre: Skipped 1 previous similar message [ 3637.216098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3637.263751] LustreError: 221509:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3637.265797] LustreError: 221509:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3637.300149] Lustre: server umount lustre-MDT0000 complete [ 3638.797431] LustreError: 218364:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086269 with bad export cookie 4759942864458049098 [ 3638.802948] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3638.877221] LustreError: 221714:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3638.880592] LustreError: 221714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3638.948447] Lustre: server umount lustre-MDT0001 complete [ 3644.618458] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3647.807306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.954576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.209961] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3651.906751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3652.031779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3653.277143] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3654.222551] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3656.373937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3658.279032] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3660.968484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3660.994957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3661.053772] Lustre: lustre-OST0001: new disk, initializing [ 3661.057509] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3661.092408] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3661.095555] Lustre: Skipped 1 previous similar message [ 3662.939511] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3664.263267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3664.267073] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3664.291969] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3668.332184] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3673.030325] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3682.762818] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:51:53 (1769086313) [ 3684.839897] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.845058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.846357] Lustre: Skipped 3 previous similar messages [ 3684.849477] Lustre: Skipped 2 previous similar messages [ 3689.874118] LustreError: 226260:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3689.877070] LustreError: 226260:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3689.929646] Lustre: server umount lustre-MDT0000 complete [ 3689.957567] LustreError: 222885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3689.966493] LustreError: 222885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3691.247363] LustreError: 223987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086321 with bad export cookie 4759942864458049987 [ 3691.250268] LustreError: 223987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3691.252305] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.366622] Lustre: server umount lustre-MDT0001 complete [ 3703.292185] LustreError: 226663:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3703.296158] LustreError: 226663:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3703.321579] Lustre: server umount lustre-OST0000 complete [ 3714.357785] LustreError: 226865:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3714.361206] LustreError: 226865:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3714.415393] Lustre: server umount lustre-OST0001 complete [ 3716.367395] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3720.415752] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3723.466685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3726.192542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3728.845869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3731.794524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3735.025057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3735.059176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.144352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3735.156215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3735.197855] Lustre: lustre-MDT0000: new disk, initializing [ 3735.224847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.232442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3736.778378] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3741.932273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3741.951339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3741.982276] Lustre: 229049:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3742.079890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3743.697781] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3747.455697] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3749.151818] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3752.242226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3752.276337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3752.399860] Lustre: lustre-OST0000: new disk, initializing [ 3752.402822] Lustre: Skipped 1 previous similar message [ 3752.406697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3752.409896] Lustre: Skipped 2 previous similar messages [ 3754.116349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3754.118506] Lustre: Skipped 1 previous similar message [ 3754.121492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3754.163631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3754.790311] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3758.320410] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3760.166884] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3760.255130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3761.949098] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3762.035890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3764.710085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3764.710273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.713349] LustreError: Skipped 1 previous similar message [ 3764.719180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.722234] Lustre: Skipped 2 previous similar messages [ 3768.887127] LustreError: 231986:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3768.889152] LustreError: 231986:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3768.932262] Lustre: server umount lustre-OST0000 complete [ 3772.897538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3772.901725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.907878] Lustre: Skipped 1 previous similar message [ 3776.568356] Lustre: server umount lustre-MDT0000 complete [ 3777.920394] LustreError: 232188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086408 with bad export cookie 4759942864458052087 [ 3777.925127] LustreError: 232188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3777.925695] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.062323] Lustre: server umount lustre-MDT0001 complete [ 3781.144256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.369112] LustreError: 232864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.409574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.413971] Lustre: Skipped 2 previous similar messages [ 3783.333858] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3786.476194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3787.913442] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3789.829576] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3791.404809] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3794.053484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3796.567454] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3798.854997] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3944.931365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3944.933021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.935852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3944.939154] Lustre: Skipped 2 previous similar messages [ 3944.942191] Lustre: Skipped 4 previous similar messages [ 3948.726768] LustreError: 235150:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3948.730014] LustreError: 235150:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3948.778113] Lustre: server umount lustre-OST0000 complete [ 3950.561198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3956.311664] Lustre: server umount lustre-MDT0000 complete [ 3957.697762] LustreError: 233585:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086588 with bad export cookie 4759942864458052969 [ 3957.702852] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3957.703386] LustreError: 233585:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3957.858989] Lustre: server umount lustre-MDT0001 complete [ 4072.771382] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4073.846182] Key type lgssc unregistered [ 4073.972488] LNet: 236122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4073.974537] LNetError: 236122:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4073.985426] LNet: Removed LNI 192.168.203.103@tcp [ 4074.283110] Key type .llcrypt unregistered [ 4074.284074] Key type ._llcrypt unregistered [ 4083.525819] Key type ._llcrypt registered [ 4083.526891] Key type .llcrypt registered [ 4083.564976] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4090.513167] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4091.018719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4091.030861] alg: No test for adler32 (adler32-zlib) [ 4091.899650] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 4092.009792] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4093.600212] Key type lgssc registered [ 4094.051874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4097.344718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4100.107793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4102.397197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4105.632894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4112.875821] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4117.220650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4117.238870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4117.244482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.350838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4118.367628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4118.423794] Lustre: lustre-MDT0000: new disk, initializing [ 4118.465530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.478478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4120.206222] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4125.460894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4125.504099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4125.551428] Lustre: 240123:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4125.595068] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4125.597642] Lustre: Skipped 1 previous similar message [ 4125.643282] Lustre: lustre-MDT0001: new disk, initializing [ 4125.685258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4125.704288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4125.710155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4127.530883] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4130.233403] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4133.179405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4133.206378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4133.319597] Lustre: lustre-OST0000: new disk, initializing [ 4133.322160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4133.363049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4135.692279] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4136.850628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4136.858261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4136.932099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4140.555073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4140.589759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4140.659479] Lustre: lustre-OST0001: new disk, initializing [ 4140.661973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4140.701737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4142.854195] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4145.697629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4145.704277] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4145.758285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4149.292890] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4152.424933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4161.506571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4161.510910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4161.518866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4163.552849] LustreError: 243646:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4163.599624] Lustre: server umount lustre-MDT0000 complete [ 4164.069799] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4164.073274] Lustre: Skipped 2 previous similar messages [ 4165.170938] LustreError: 240903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086795 with bad export cookie 11747554330875724850 [ 4165.178143] LustreError: 240903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4165.180440] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4167.305357] LustreError: 243849:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4167.309180] LustreError: 243849:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4167.406641] Lustre: server umount lustre-MDT0001 complete [ 4178.429105] LustreError: 244049:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4178.432388] LustreError: 244049:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4178.456956] Lustre: server umount lustre-OST0000 complete [ 4189.949122] LustreError: 244251:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4189.953772] LustreError: 244251:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4190.029488] Lustre: server umount lustre-OST0001 complete [ 4196.297852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4199.961859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4200.353292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4203.906086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4207.496498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4207.908108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4215.349382] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4219.755827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.767175] Lustre: lustre-MDT0000: reset Object Index mappings [ 4219.968222] LustreError: 247559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.975310] LustreError: 247559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4219.989626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.504988] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4224.993028] LustreError: 247559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.148967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4225.155799] Lustre: lustre-MDT0001: reset Object Index mappings [ 4226.821777] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4228.015239] Lustre: 248669:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4230.774789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4230.964800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.968683] Lustre: Skipped 1 previous similar message [ 4233.553736] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4236.074196] LustreError: 249038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.244339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4239.848432] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4244.525567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4244.532890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4248.326938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4249.661816] Lustre: 250528:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4256.224887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4256.227345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.231347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.814923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.815314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.819521] Lustre: Skipped 1 previous similar message [ 4259.826673] Lustre: Skipped 1 previous similar message [ 4262.098208] LustreError: 250633:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4262.102191] LustreError: 250633:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4262.143684] Lustre: server umount lustre-MDT0000 complete [ 4263.794579] LustreError: 248287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086894 with bad export cookie 11747554330875731626 [ 4263.799618] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4263.918336] Lustre: server umount lustre-MDT0001 complete [ 4275.708944] LustreError: 251036:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4275.711602] LustreError: 251036:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4275.737468] Lustre: server umount lustre-OST0000 complete [ 4287.621825] Lustre: server umount lustre-OST0001 complete [ 4295.932736] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 08:02:06 (1769086926) [ 4303.133902] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4307.649317] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4311.857817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4314.676753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4317.116692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4319.943148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4323.091421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4323.124037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.238708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4323.254168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4323.297668] Lustre: lustre-MDT0000: new disk, initializing [ 4323.329463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.331460] Lustre: Skipped 1 previous similar message [ 4323.338445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4325.275938] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4330.190092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4330.234239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4330.288716] Lustre: 254756:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4330.444522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4332.312108] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4336.483356] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4338.330072] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4341.789932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4341.825846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4341.948385] Lustre: lustre-OST0000: new disk, initializing [ 4341.950244] Lustre: Skipped 1 previous similar message [ 4341.952560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4341.954784] Lustre: Skipped 2 previous similar messages [ 4343.321423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4343.326334] Lustre: Skipped 1 previous similar message [ 4343.329905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4343.383782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4344.623789] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4348.757152] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4350.757317] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4350.893792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4352.829723] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4352.936306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4358.632424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4358.632698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.637067] LustreError: Skipped 1 previous similar message [ 4358.646691] Lustre: Skipped 1 previous similar message [ 4358.650906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4358.656181] Lustre: Skipped 3 previous similar messages [ 4363.749757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4363.753378] Lustre: Skipped 1 previous similar message [ 4368.352177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4368.380197] LustreError: 257694:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4368.383375] LustreError: 257694:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4368.447111] Lustre: server umount lustre-OST0000 complete [ 4371.426135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4371.431680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.987403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.991106] Lustre: Skipped 2 previous similar messages [ 4376.585404] Lustre: server umount lustre-MDT0000 complete [ 4378.452390] LustreError: 255536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769087009 with bad export cookie 11747554330875733719 [ 4378.459867] LustreError: 255536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4378.461157] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.630015] Lustre: server umount lustre-MDT0001 complete [ 4387.787545] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4392.478635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.676066] LustreError: 259272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.696104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.700937] Lustre: Skipped 2 previous similar messages [ 4394.595662] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4398.050342] LustreError: 259272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.662990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4400.460406] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4401.911953] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4405.262745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4408.342870] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4412.629073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4412.680271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4412.786516] Lustre: lustre-OST0001: new disk, initializing [ 4412.790142] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4413.928895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4413.935539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4413.999376] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4415.633906] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4420.896502] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4422.736078] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4430.324236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4430.329518] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4433.381579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.382673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4433.387660] Lustre: Skipped 3 previous similar messages [ 4433.392085] Lustre: Skipped 3 previous similar messages [ 4438.809536] LustreError: 262780:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4438.813750] LustreError: 262780:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4438.854903] Lustre: server umount lustre-MDT0000 complete [ 4440.644934] LustreError: 259252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769087071 with bad export cookie 11747554330875734601 [ 4440.651711] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4440.652461] LustreError: 259252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4440.823715] Lustre: server umount lustre-MDT0001 complete [ 4452.887808] Lustre: server umount lustre-OST0000 complete [ 4464.968645] Lustre: server umount lustre-OST0001 complete [ 4472.491728] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4476.845273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.034175] LustreError: 264558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.058099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.060304] Lustre: Skipped 3 previous similar messages [ 4478.614915] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4482.754139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4484.459152] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4485.827776] Lustre: 265667:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4489.053675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4491.866281] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4496.220256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4499.035319] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4501.870420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4501.878490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4508.755866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4510.673342] Lustre: 267524:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4518.227105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4518.233600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4522.946312] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 08:05:53 (1769087153) [ 4537.830984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.835238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4537.840848] Lustre: Skipped 4 previous similar messages [ 4537.845125] Lustre: Skipped 6 previous similar messages [ 4541.138914] LustreError: 268231:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4541.143146] LustreError: 268231:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4541.184732] Lustre: server umount lustre-MDT0000 complete [ 4542.928857] LustreError: 267526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769087173 with bad export cookie 11747554330875740873 [ 4542.934863] LustreError: 267526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4542.934999] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4542.947313] LustreError: 264554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.955394] LustreError: 264554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4557.280121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4557.403248] Lustre: server umount lustre-MDT0001 complete [ 4565.322564] Lustre: server umount lustre-OST0000 complete [ 4573.300357] Lustre: server umount lustre-OST0001 complete [ 4576.271955] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4580.937141] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4586.108801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4589.384429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4592.269367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4595.749366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4603.766614] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4608.864425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4608.902439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4609.054322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4609.072405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4609.135505] Lustre: lustre-MDT0000: new disk, initializing [ 4609.178630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.182992] Lustre: Skipped 3 previous similar messages [ 4609.193924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4611.148275] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4617.002452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4617.047333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4617.101553] Lustre: 271730:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4617.263705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4619.255917] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4625.632643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4625.675763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4627.491643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4628.691774] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4634.585483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4634.628083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4636.423879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4637.171360] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4642.855492] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4644.724562] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4644.729771] Lustre: Skipped 1 previous similar message [ 4656.089775] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 08:08:06 (1769087286) [ 4658.147083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4658.152536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4658.160874] Lustre: Skipped 2 previous similar messages [ 4658.164860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4658.169018] Lustre: Skipped 6 previous similar messages [ 4663.311354] Lustre: server umount lustre-MDT0000 complete [ 4665.127065] LustreError: 273331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769087295 with bad export cookie 11747554330875748734 [ 4665.133423] LustreError: 273331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4665.134729] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4665.290720] Lustre: server umount lustre-MDT0001 complete [ 4676.415226] LustreError: 275740:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4676.419123] LustreError: 275740:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 4676.446465] Lustre: server umount lustre-OST0000 complete [ 4688.636157] Lustre: server umount lustre-OST0001 complete [ 4697.124405] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4701.839758] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4706.853319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4710.321459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4713.640562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4716.973536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4720.598802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4720.633865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.759164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4720.776285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4720.780159] Lustre: Skipped 4 previous similar messages [ 4720.836345] Lustre: lustre-MDT0000: new disk, initializing [ 4720.837859] Lustre: Skipped 3 previous similar messages [ 4720.875866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4720.880758] Lustre: Skipped 3 previous similar messages [ 4722.667231] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4728.239893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4728.282091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4728.335628] Lustre: 278784:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4728.512838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4728.516358] Lustre: Skipped 2 previous similar messages [ 4730.297716] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4734.239947] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4736.038609] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4739.402450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4739.444688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4740.754303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4742.140343] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4746.162558] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4746.909799] Lustre: Mounted lustre-client [ 4747.234403] LustreError: 281290:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff88e76e2f9800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4747.269371] Lustre: Unmounted lustre-client [ 4751.330112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4751.334933] LustreError: Skipped 1 previous similar message [ 4756.785019] Lustre: server umount lustre-OST0000 complete [ 4764.550809] Lustre: server umount lustre-MDT0000 complete [ 4766.419614] LustreError: 278777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769087397 with bad export cookie 11747554330875750253 [ 4766.428438] LustreError: 278777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4766.428857] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4766.615340] Lustre: server umount lustre-MDT0001 complete [ 4771.250973] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4772.384127] Key type lgssc unregistered [ 4799.136481] LNet: 282565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4799.140541] LNetError: 282565:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4875.942673] LNet: Removed LNI 192.168.203.103@tcp [ 4876.405163] Key type .llcrypt unregistered [ 4876.407546] Key type ._llcrypt unregistered [ 4883.089948] Key type ._llcrypt registered [ 4883.091042] Key type .llcrypt registered [ 4883.121542] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4890.792195] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4891.113876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4891.157427] alg: No test for adler32 (adler32-zlib) [ 4891.994937] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 4892.067727] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4893.640171] Key type lgssc registered [ 4894.213380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4898.462075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4901.647082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4904.746568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4908.016831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4908.989884] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 08:12:19 (1769087539) [ 4912.074698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4912.097861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4912.109913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.225324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4913.244158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4913.303689] Lustre: lustre-MDT0000: new disk, initializing [ 4913.344144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.356967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4914.915076] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4920.247820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4920.285748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4920.343105] Lustre: 285738:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4920.385586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4920.389405] Lustre: Skipped 1 previous similar message [ 4920.438232] Lustre: lustre-MDT0001: new disk, initializing [ 4920.479512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4920.496810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4920.501765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4922.322991] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4926.255976] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4927.895169] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4930.973306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4931.011949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4931.151974] Lustre: lustre-OST0000: new disk, initializing [ 4931.155823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4931.204433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.750336] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4936.204305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4936.210506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4936.266263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4937.724712] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4946.403399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4946.408284] LustreError: Skipped 1 previous similar message [ 4946.408320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.412186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4946.418642] Lustre: Skipped 1 previous similar message [ 4948.467125] LustreError: 288183:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4948.522864] Lustre: server umount lustre-OST0000 complete [ 4951.523922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.524476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4951.525353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4951.525362] Lustre: Skipped 1 previous similar message [ 4951.531256] Lustre: Skipped 1 previous similar message [ 4956.492306] LustreError: 288385:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4956.495812] LustreError: 288385:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4956.532425] Lustre: server umount lustre-MDT0000 complete [ 4958.335094] LustreError: 287497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769087589 with bad export cookie 6954665426724402623 [ 4958.341793] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4958.342113] LustreError: 287497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4958.436799] LustreError: 288587:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4958.440152] LustreError: 288587:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4958.501314] Lustre: server umount lustre-MDT0001 complete [ 4963.351624] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4964.756129] Key type lgssc unregistered [ 4964.933689] LNet: 289118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4964.938524] LNetError: 289118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4964.950601] LNet: Removed LNI 192.168.203.103@tcp [ 4965.403365] Key type .llcrypt unregistered [ 4965.405584] Key type ._llcrypt unregistered [ 5009.892438] Key type ._llcrypt registered [ 5009.894153] Key type .llcrypt registered [ 5009.941237] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5010.088227] Key type .llcrypt unregistered [ 5010.090046] Key type ._llcrypt unregistered [ 5046.901067] Key type ._llcrypt registered [ 5046.903165] Key type .llcrypt registered [ 5046.954870] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5047.102891] Key type .llcrypt unregistered [ 5047.105063] Key type ._llcrypt unregistered [ 5062.062833] Key type ._llcrypt registered [ 5062.064229] Key type .llcrypt registered [ 5062.115787] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5062.263429] Key type .llcrypt unregistered [ 5062.265466] Key type ._llcrypt unregistered [ 5076.687892] Key type ._llcrypt registered [ 5076.689010] Key type .llcrypt registered [ 5076.727489] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5076.825230] Key type .llcrypt unregistered [ 5076.826379] Key type ._llcrypt unregistered [ 5110.059557] Key type ._llcrypt registered [ 5110.060755] Key type .llcrypt registered [ 5110.101356] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5110.202267] Key type .llcrypt unregistered [ 5110.204088] Key type ._llcrypt unregistered [ 5123.478961] Key type ._llcrypt registered [ 5123.480590] Key type .llcrypt registered [ 5123.525039] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5123.642379] Key type .llcrypt unregistered [ 5123.644147] Key type ._llcrypt unregistered [ 5137.011470] Key type ._llcrypt registered [ 5137.013406] Key type .llcrypt registered [ 5137.072687] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5137.187293] Key type .llcrypt unregistered [ 5137.190047] Key type ._llcrypt unregistered [ 5151.130455] Key type ._llcrypt registered [ 5151.131527] Key type .llcrypt registered [ 5151.165819] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5151.253195] Key type .llcrypt unregistered [ 5151.254203] Key type ._llcrypt unregistered [ 5164.302879] Key type ._llcrypt registered [ 5164.304103] Key type .llcrypt registered [ 5164.353118] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5164.471580] Key type .llcrypt unregistered [ 5164.472922] Key type ._llcrypt unregistered [ 5178.259727] Key type ._llcrypt registered [ 5178.261331] Key type .llcrypt registered [ 5178.318769] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5178.457319] Key type .llcrypt unregistered [ 5178.458470] Key type ._llcrypt unregistered [ 5206.220296] Key type ._llcrypt registered [ 5206.222058] Key type .llcrypt registered [ 5206.275626] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5206.428319] Key type .llcrypt unregistered [ 5206.430620] Key type ._llcrypt unregistered [ 5219.871085] Key type ._llcrypt registered [ 5219.872950] Key type .llcrypt registered [ 5219.935000] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5220.088329] Key type .llcrypt unregistered [ 5220.089968] Key type ._llcrypt unregistered [ 5233.274233] Key type ._llcrypt registered [ 5233.275245] Key type .llcrypt registered [ 5233.310809] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5233.412171] Key type .llcrypt unregistered [ 5233.413212] Key type ._llcrypt unregistered [ 5256.606623] Key type ._llcrypt registered [ 5256.607671] Key type .llcrypt registered [ 5256.646644] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 5263.925850] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5264.350648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5264.401853] alg: No test for adler32 (adler32-zlib) [ 5265.298626] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 5265.426902] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5267.032212] Key type lgssc registered [ 5267.624257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5272.040557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5275.352056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5278.465117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5281.574200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5282.463766] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5172 sec ========== 08:18:33 (1769087913) [ 5283.241076] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:18:33 (1769087913) === [ 5284.710460] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:18:35 (1769087915) === [ 5302.129045] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5303.427366] Key type lgssc unregistered [ 5303.576534] LNet: 310821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5303.580480] LNetError: 310821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5303.592630] LNet: Removed LNI 192.168.203.103@tcp [ 5303.959696] Key type .llcrypt unregistered [ 5303.961440] Key type ._llcrypt unregistered