[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 560935597 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002236] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007404] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011017] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022464] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027419] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029078] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030080] *** VALIDATE tmpfs *** [ 0.031300] *** VALIDATE proc *** [ 0.032170] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.033820] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034123] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036021] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.039921] debug: unmapping init [mem 0xffffffff94459000-0xffffffff94460fff] [ 0.041158] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042545] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043021] ... version: 2 [ 0.044012] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.045995] ... value mask: 0000ffffffffffff [ 0.046012] ... max period: 00007fffffffffff [ 0.046976] ... fixed-purpose events: 3 [ 0.047011] ... event mask: 000000070000000f [ 0.048220] rcu: Hierarchical SRCU implementation. [ 0.050098] smp: Bringing up secondary CPUs ... [ 0.051610] x86: Booting SMP configuration: [ 0.052034] .... node #0, CPUs: #1 #2 #3 [ 0.055234] smp: Brought up 1 node, 4 CPUs [ 0.057013] smpboot: Max logical packages: 1 [ 0.058015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.184204] node 0 deferred pages initialised in 124ms [ 0.188177] devtmpfs: initialized [ 0.189263] x86/mm: Memory block size: 128MB [ 0.192833] gcov: version magic: 0x41383552 [ 0.196378] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.197097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.199315] pinctrl core: initialized pinctrl subsystem [ 0.203249] [ 0.203862] ************************************************************* [ 0.206013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209015] ** ** [ 0.211012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.214017] ** ** [ 0.216014] ** This means that this kernel is built to expose internal ** [ 0.219017] ** IOMMU data structures, which may compromise security on ** [ 0.222014] ** your system. ** [ 0.224011] ** ** [ 0.226017] ** If you see this message and you are not debugging the ** [ 0.229014] ** kernel, report this immediately to your vendor! ** [ 0.231013] ** ** [ 0.234014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236014] ************************************************************* [ 0.238715] NET: Registered protocol family 16 [ 0.240401] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.242068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.245086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.249034] cpuidle: using governor menu [ 0.250930] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.253503] PCI: Using configuration type 1 for base access [ 0.255021] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.264062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.271062] cryptd: max_cpu_qlen set to 1000 [ 0.273218] ACPI: Added _OSI(Module Device) [ 0.274020] ACPI: Added _OSI(Processor Device) [ 0.275047] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.277038] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280770] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.287066] ACPI: Interpreter enabled [ 0.288063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290016] ACPI: Using IOAPIC for interrupt routing [ 0.291115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.294444] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304484] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.306048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.308026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.310066] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.314095] acpiphp: Slot [2] registered [ 0.315097] acpiphp: Slot [5] registered [ 0.316120] acpiphp: Slot [6] registered [ 0.317067] acpiphp: Slot [7] registered [ 0.318032] acpiphp: Slot [8] registered [ 0.319045] acpiphp: Slot [9] registered [ 0.320014] acpiphp: Slot [10] registered [ 0.320978] acpiphp: Slot [3] registered [ 0.321056] acpiphp: Slot [4] registered [ 0.323094] acpiphp: Slot [11] registered [ 0.324085] acpiphp: Slot [12] registered [ 0.325076] acpiphp: Slot [13] registered [ 0.326080] acpiphp: Slot [14] registered [ 0.327074] acpiphp: Slot [15] registered [ 0.328063] acpiphp: Slot [16] registered [ 0.328987] acpiphp: Slot [17] registered [ 0.330107] acpiphp: Slot [18] registered [ 0.331090] acpiphp: Slot [19] registered [ 0.332078] acpiphp: Slot [20] registered [ 0.333090] acpiphp: Slot [21] registered [ 0.335097] acpiphp: Slot [22] registered [ 0.336075] acpiphp: Slot [23] registered [ 0.337121] acpiphp: Slot [24] registered [ 0.339127] acpiphp: Slot [25] registered [ 0.341070] acpiphp: Slot [26] registered [ 0.342087] acpiphp: Slot [27] registered [ 0.343128] acpiphp: Slot [28] registered [ 0.344095] acpiphp: Slot [29] registered [ 0.346098] acpiphp: Slot [30] registered [ 0.347075] acpiphp: Slot [31] registered [ 0.348042] PCI host bridge to bus 0000:00 [ 0.349017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.351019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.352018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.355026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.357027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.360030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.362207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.365199] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368412] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378890] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.384010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.386021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.388030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.392033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.395715] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.398618] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.400038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402868] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407087] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.419018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.425031] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.431460] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.453022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.485915] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.492021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.497018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.518824] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.523012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.532027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.556339] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.565019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.573018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.594021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.605592] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.610018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.616019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641125] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647117] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.653017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.677028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.689552] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.691205] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.692446] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.694318] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.696208] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.701076] iommu: Default domain type: Passthrough [ 0.703542] SCSI subsystem initialized [ 0.705139] ACPI: bus type USB registered [ 0.706131] usbcore: registered new interface driver usbfs [ 0.708110] usbcore: registered new interface driver hub [ 0.710087] usbcore: registered new device driver usb [ 0.712180] pps_core: LinuxPPS API ver. 1 registered [ 0.714012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.717066] PTP clock support registered [ 0.720136] EDAC MC: Ver: 3.0.0 [ 0.722138] PCI: Using ACPI for IRQ routing [ 0.723964] NetLabel: Initializing [ 0.726015] NetLabel: domain hash size = 128 [ 0.727011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729083] NetLabel: unlabeled traffic allowed by default [ 0.731114] vgaarb: loaded [ 0.733286] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.735018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741004] clocksource: Switched to clocksource kvm-clock [ 0.857718] VFS: Disk quotas dquot_6.6.0 [ 0.859105] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.861580] *** VALIDATE ramfs *** [ 0.862885] *** VALIDATE hugetlbfs *** [ 0.864573] pnp: PnP ACPI init [ 0.867042] pnp: PnP ACPI: found 6 devices [ 0.882032] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.885124] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.886829] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.888112] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.889462] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.891479] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.894147] NET: Registered protocol family 2 [ 0.896190] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.899915] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.903177] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.908393] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.911969] TCP: Hash tables configured (established 65536 bind 65536) [ 0.914961] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.918130] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920766] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923391] NET: Registered protocol family 1 [ 0.926566] RPC: Registered named UNIX socket transport module. [ 0.928821] RPC: Registered udp transport module. [ 0.930590] RPC: Registered tcp transport module. [ 0.931895] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.933321] NET: Registered protocol family 44 [ 0.934112] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.935877] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.937455] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.938846] PCI: CLS 0 bytes, default 64 [ 0.940158] Unpacking initramfs... [ 2.371760] debug: unmapping init [mem 0xffff9d31bcc54000-0xffff9d31bffbffff] [ 2.374581] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.376205] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.379151] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.963146] Initialise system trusted keyrings [ 2.964726] Key type blacklist registered [ 2.969988] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.981921] zbud: loaded [ 2.985862] *** VALIDATE nfs *** [ 2.987298] *** VALIDATE nfs4 *** [ 2.989030] pstore: using deflate compression [ 2.993420] Platform Keyring initialized [ 3.138051] NET: Registered protocol family 38 [ 3.139489] Key type asymmetric registered [ 3.141152] Asymmetric key parser 'x509' registered [ 3.143145] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.146653] io scheduler mq-deadline registered [ 3.148421] io scheduler kyber registered [ 3.150136] io scheduler bfq registered [ 3.152264] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.155581] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.158257] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.161364] ACPI: Power Button [PWRF] [ 3.168442] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.177579] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.189181] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.200306] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.217407] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.244034] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.272972] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.280669] Non-volatile memory driver v1.3 [ 3.282528] Linux agpgart interface v0.103 [ 3.318241] virtio_blk virtio1: [vda] 139368 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.322617] vda: detected capacity change from 0 to 71356416 [ 3.374603] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.377351] vdb: detected capacity change from 0 to 1073741824 [ 3.405301] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.408848] vdc: detected capacity change from 0 to 2621440000 [ 3.435648] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.437880] vdd: detected capacity change from 0 to 2621440000 [ 3.458637] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.461716] vde: detected capacity change from 0 to 4294967296 [ 3.474228] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.475955] vdf: detected capacity change from 0 to 4294967296 [ 3.480216] libphy: Fixed MDIO Bus: probed [ 3.490533] usbcore: registered new interface driver usbserial_generic [ 3.493079] usbserial: USB Serial support registered for generic [ 3.496259] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.499902] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.501153] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.502808] mousedev: PS/2 mouse device common for all mice [ 3.505900] rtc_cmos 00:05: RTC can wake from S4 [ 3.505931] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.510825] rtc_cmos 00:05: registered as rtc0 [ 3.513031] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.517035] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.517081] intel_pstate: CPU model not supported [ 3.522847] hid: raw HID events driver (C) Jiri Kosina [ 3.525964] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.526892] usbcore: registered new interface driver usbhid [ 3.534796] usbhid: USB HID core driver [ 3.536683] drop_monitor: Initializing network drop monitor service [ 3.540315] Initializing XFRM netlink socket [ 3.542455] NET: Registered protocol family 10 [ 3.546610] Segment Routing with IPv6 [ 3.547879] NET: Registered protocol family 17 [ 3.550165] mpls_gso: MPLS GSO support [ 3.555472] RAS: Correctable Errors collector initialized. [ 3.557902] AVX version of gcm_enc/dec engaged. [ 3.559750] AES CTR mode by8 optimization enabled [ 3.658209] sched_clock: Marking stable (3658183662, 0)->(4488888830, -830705168) [ 3.662753] registered taskstats version 1 [ 3.665303] Loading compiled-in X.509 certificates [ 3.669719] zswap: loaded using pool lzo/zbud [ 3.704597] Key type big_key registered [ 3.754291] Key type encrypted registered [ 3.755819] ima: No TPM chip found, activating TPM-bypass! [ 3.758066] ima: Allocated hash algorithm: sha1 [ 3.761389] ima: No architecture policies found [ 3.763405] evm: Initialising EVM extended attributes: [ 3.765518] evm: security.selinux [ 3.766804] evm: security.ima [ 3.768257] evm: security.capability [ 3.769709] evm: HMAC attrs: 0x1 [ 3.773815] rtc_cmos 00:05: setting system clock to 2026-02-10 00:44:01 UTC (1770684241) [ 3.780941] debug: unmapping init [mem 0xffffffff95403000-0xffffffff955fffff] [ 3.784647] debug: unmapping init [mem 0xffffffff94182000-0xffffffff94458fff] [ 3.793099] Write protecting the kernel read-only data: 28672k [ 3.796807] debug: unmapping init [mem 0xffffffff92803000-0xffffffff929fffff] [ 3.800431] debug: unmapping init [mem 0xffffffff93114000-0xffffffff931fffff] [ 3.850210] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.859263] systemd[1]: Detected virtualization kvm. [ 3.861287] systemd[1]: Detected architecture x86-64. [ 3.863542] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.897436] systemd[1]: No hostname configured. [ 3.899528] systemd[1]: Set hostname to . [ 3.902302] random: systemd: uninitialized urandom read (16 bytes read) [ 3.905237] systemd[1]: Initializing machine ID from random generator. [ 3.972708] random: ln: uninitialized urandom read (6 bytes read) [ 4.080702] random: systemd: uninitialized urandom read (16 bytes read) [ 4.083756] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.090774] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.100904] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.851306] device-mapper: uevent: version 1.0.3 [ 4.853376] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.662414] random: fast init done [ 5.835291] virtio_net virtio0 ens2: renamed from eth0 [ 5.850718] scsi host0: ata_piix [ 5.919133] scsi host1: ata_piix [ 5.920551] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.922899] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.473606] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.277393] random: crng init done [ 10.279130] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.849284] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.098088] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.372453] SELinux: Disabled at runtime. [ 12.424964] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.431530] systemd[1]: Detected virtualization kvm. [ 12.432773] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.934336] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.938316] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.943651] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.947538] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.951645] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.962797] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.971227] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK [ 13.049795] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] [ 13.490264] squashfs: version 4.0 (2009/01/31) Phillip Lougher Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.799670] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.808843] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.007989] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.023842] EDAC sbridge: Ver: 1.1.2 [ 15.760091] Key type dns_resolver registered [ 16.089564] NFS: Registering the id_resolver key type [ 16.091706] Key type id_resolver registered [ 16.093565] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg459-server login: [ 53.283937] libcfs: loading out-of-tree module taints kernel. [ 53.420387] Key type ._llcrypt registered [ 53.422873] Key type .llcrypt registered [ 53.586572] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 74.784268] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 76.737959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.752111] alg: No test for adler32 (adler32-zlib) [ 78.145584] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 79.030050] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 80.895177] Key type lgssc registered [ 82.571161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.814257] hrtimer: interrupt took 5248287 ns [ 109.333151] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 159.987542] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 176.288736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 176.335874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.714424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.787860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.957943] Lustre: lustre-MDT0000: new disk, initializing [ 178.085274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.104825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.249510] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.037295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 200.213547] Lustre: 6514:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.283903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.289177] Lustre: Skipped 1 previous similar message [ 200.482452] Lustre: lustre-MDT0001: new disk, initializing [ 200.614450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.666092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.673349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.003740] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.918486] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.861680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 225.406503] Lustre: lustre-OST0000: new disk, initializing [ 225.420241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.432569] Lustre: 8419:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.553963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.008675] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.559927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.578250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.676452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.262596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 254.412966] Lustre: lustre-OST0001: new disk, initializing [ 254.416808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.424266] Lustre: 9476:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.562078] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 261.194508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.204747] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.354412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.871739] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.405670] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.177156] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.967932] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.456956] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing yml_node [ 307.840433] Lustre: DEBUG MARKER: Client: 2.17.50.65 [ 310.970188] Lustre: DEBUG MARKER: MDS: 2.17.50.65 [ 314.120251] Lustre: DEBUG MARKER: OSS: 2.17.50.65 [ 316.084368] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Feb 9 19:49:12 EST 2026 [ 338.704936] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 340.413485] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 358.884282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.885384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.911991] Lustre: Skipped 2 previous similar messages [ 358.928724] Lustre: Skipped 3 previous similar messages [ 362.313217] LustreError: 12228:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 362.425167] Lustre: server umount lustre-MDT0000 complete [ 367.402134] LustreError: 6507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770684605 with bad export cookie 14088837382265073964 [ 367.431791] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.438329] LustreError: 6507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 369.121820] LustreError: 9994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.122642] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.132779] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.148298] LustreError: 9994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 369.190638] Lustre: Skipped 2 previous similar messages [ 372.193947] LustreError: 12108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.206114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.230731] LustreError: 12108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 372.249576] Lustre: Skipped 2 previous similar messages [ 373.768648] LustreError: 12429:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 373.777876] LustreError: 12429:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 374.565900] Lustre: server umount lustre-MDT0001 complete [ 379.411222] LustreError: 12631:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 379.414585] LustreError: 12631:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 379.768408] Lustre: server umount lustre-OST0000 complete [ 384.962845] LustreError: 12832:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 384.972682] LustreError: 12832:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 385.228108] Lustre: server umount lustre-OST0001 complete [ 393.792164] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 404.906351] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 450.199885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.558748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 450.588414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.654300] Lustre: lustre-MDT0000: new disk, initializing [ 450.770337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.807555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 456.049634] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 468.449534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 468.582518] Lustre: 15004:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 468.651070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 468.655560] Lustre: Skipped 1 previous similar message [ 468.752542] Lustre: lustre-MDT0001: new disk, initializing [ 468.877459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 468.934983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 468.945113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 474.045161] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 483.019994] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 487.701442] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 497.488944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 498.046522] Lustre: lustre-OST0000: new disk, initializing [ 498.053530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 498.067314] Lustre: 16747:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 498.203953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.722667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 499.742809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 499.910929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 507.504747] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 517.766940] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 522.855781] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 523.136499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 529.193863] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 529.480789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 535.527455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.557452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 535.575343] Lustre: Skipped 1 previous similar message [ 538.392193] LustreError: 17939:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 538.400204] LustreError: 17939:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 538.764201] Lustre: server umount lustre-OST0000 complete [ 545.771094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.778403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 545.791707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.796460] Lustre: Skipped 2 previous similar messages [ 549.905164] LustreError: 18141:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 549.923766] LustreError: 18141:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 550.112892] Lustre: server umount lustre-MDT0000 complete [ 550.881845] LustreError: 16450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.891994] LustreError: 16450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 554.628081] LustreError: 14998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770684792 with bad export cookie 14088837382265075497 [ 554.638412] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.641355] LustreError: 14998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 554.984348] Lustre: server umount lustre-MDT0001 complete [ 562.031693] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:53:18 (1770684798) [ 583.209131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.012079] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.161826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.287561] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.370660] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 598.501916] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.523240] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 599.930505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 605.304960] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 612.019969] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.457444] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 620.277450] Lustre: Failing over lustre-MDT0000 [ 620.541536] LustreError: 20819:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 620.553419] LustreError: 20819:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 620.645196] Lustre: server umount lustre-MDT0000 complete [ 621.024651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 621.028202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.035347] LustreError: 20189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.042024] Lustre: Skipped 1 previous similar message [ 621.094859] LustreError: 20189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 636.383442] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684858/real 1770684858] req@ffff9d322aecf100 x1856697077503872/t0(0) o400->MGC192.168.204.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 636.407104] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.260675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.390050] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 652.265392] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 652.266556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 652.309037] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 652.530842] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 661.839664] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 667.618274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 667.629453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.672490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 667.675850] Lustre: Skipped 1 previous similar message [ 673.364584] LustreError: 21798:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 673.377679] LustreError: 21798:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 673.745781] Lustre: server umount lustre-MDT0000 complete [ 677.876107] LustreError: 19477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.890928] LustreError: 19477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 678.031666] LustreError: 19459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770684915 with bad export cookie 14088837382265076911 [ 678.034437] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 678.048854] LustreError: 19459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 678.771839] Lustre: server umount lustre-MDT0001 complete [ 689.721991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.616426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.623836] Lustre: Skipped 2 previous similar messages [ 697.769381] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 709.136739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 715.181064] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 722.172621] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 727.090801] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 729.782058] Lustre: Failing over lustre-MDT0000 [ 730.019944] Lustre: server umount lustre-MDT0000 complete [ 730.082218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.090966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 730.100426] Lustre: Skipped 1 previous similar message [ 745.441671] Lustre: 3663:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684967/real 1770684967] req@ffff9d310729c380 x1856697077533824/t0(0) o400->MGC192.168.204.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 745.516716] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.530755] LustreError: 22480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.546940] LustreError: 22480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 749.733525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.011803] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 760.684969] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 761.317652] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 761.351475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 761.358121] Lustre: Skipped 1 previous similar message [ 761.373962] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 769.606904] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 775.648520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.658431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.662052] Lustre: Skipped 2 previous similar messages [ 775.676545] Lustre: Skipped 5 previous similar messages [ 780.792506] LustreError: 24807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 780.797200] LustreError: 24807:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 781.101088] Lustre: server umount lustre-MDT0000 complete [ 784.977836] LustreError: 22459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685022 with bad export cookie 14088837382265077842 [ 784.979801] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.988549] LustreError: 22459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 785.289804] Lustre: server umount lustre-MDT0001 complete [ 796.137131] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:57:12 (1770685032) [ 818.440176] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 832.153340] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 880.213566] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 893.619558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.814315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 893.867956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 893.967026] Lustre: lustre-MDT0000: new disk, initializing [ 894.082704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.098112] Lustre: Skipped 2 previous similar messages [ 894.138548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 900.223542] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 914.987821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 915.263503] Lustre: 28485:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 915.597226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 920.517857] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 929.195697] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 934.537665] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 943.433266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 944.082994] Lustre: 30228:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 945.871080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 945.965724] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 946.094634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 946.140473] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 952.205820] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 961.152860] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 967.107297] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 997.344505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 997.351675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.367360] Lustre: Skipped 1 previous similar message [ 997.371054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 997.395376] Lustre: Skipped 3 previous similar messages [ 998.509682] LustreError: 31142:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 998.514628] LustreError: 31142:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 998.645730] Lustre: server umount lustre-OST0000 complete [ 1008.897594] Lustre: server umount lustre-MDT0000 complete [ 1012.709792] LustreError: 29309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.727248] LustreError: 29309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1013.057299] LustreError: 28475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685250 with bad export cookie 14088837382265078241 [ 1013.060762] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1013.068242] LustreError: 28475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1013.700727] Lustre: server umount lustre-MDT0001 complete [ 1027.345733] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1030.667979] Key type lgssc unregistered [ 1031.000816] LNet: 32092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.006839] LNetError: 32092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.016322] LNet: Removed LNI 192.168.204.159@tcp [ 1032.228156] Key type .llcrypt unregistered [ 1032.229887] Key type ._llcrypt unregistered [ 1061.560465] Key type ._llcrypt registered [ 1061.562151] Key type .llcrypt registered [ 1061.682600] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1063.096461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1063.311635] alg: No test for adler32 (adler32-zlib) [ 1064.523733] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1064.857935] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1066.615268] Key type lgssc registered [ 1068.322597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1078.346506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1078.360269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.012827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.350887] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1089.519343] LustreError: 33487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.531336] LustreError: 33487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1094.631186] LustreError: 33487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.123310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1097.598611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1102.830305] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1109.684781] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.473925] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1122.631323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1123.269927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.304896] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1130.750729] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1137.918342] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1154.083368] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:03:10 (1770685390) [ 1156.394546] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1158.529673] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1167.311901] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:03:23 (1770685403) [ 1177.864524] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:03:33 (1770685413) [ 1179.361419] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1191.845441] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1191.848307] Lustre: Skipped 2 previous similar messages [ 1193.893503] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:03:50 (1770685430) [ 1196.159863] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1196.173034] Lustre: Skipped 2 previous similar messages [ 1204.920969] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1204.928279] Lustre: Skipped 1 previous similar message [ 1221.319586] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1221.332413] Lustre: Skipped 5 previous similar messages [ 1224.930511] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:04:21 (1770685461) [ 1228.499758] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1232.112329] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1243.238570] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:04:39 (1770685479) [ 1258.722819] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1258.730392] Lustre: Skipped 7 previous similar messages [ 1260.647189] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:04:57 (1770685497) [ 1277.921125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1277.928450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.946552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1278.949858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1278.962233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1278.969715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.011505] Lustre: Skipped 1 previous similar message [ 1284.070412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1284.076571] Lustre: Skipped 1 previous similar message [ 1289.190549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.197321] Lustre: Skipped 1 previous similar message [ 1291.232256] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1291.308356] LustreError: 40057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1291.595596] Lustre: server umount lustre-OST0000 complete [ 1297.378712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1297.387407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.404966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.448081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.249324] LustreError: 40260:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1302.251480] LustreError: 40260:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1302.432977] Lustre: server umount lustre-MDT0000 complete [ 1304.546115] LustreError: 33487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.572861] LustreError: 33487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1306.663753] LustreError: 33467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685544 with bad export cookie 6539916414061470719 [ 1306.669469] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.679633] LustreError: 33467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1306.839987] LustreError: 40461:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1306.849921] LustreError: 40461:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1307.066973] Lustre: server umount lustre-MDT0001 complete [ 1320.525738] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1323.488967] Key type lgssc unregistered [ 1323.784083] LNet: 40994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.793753] LNetError: 40994:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.813441] LNet: Removed LNI 192.168.204.159@tcp [ 1324.949415] Key type .llcrypt unregistered [ 1324.954585] Key type ._llcrypt unregistered [ 1344.152920] Key type ._llcrypt registered [ 1344.155869] Key type .llcrypt registered [ 1344.284542] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 1361.682308] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1362.788376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1362.872132] alg: No test for adler32 (adler32-zlib) [ 1364.402546] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1364.772521] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1366.464202] Key type lgssc registered [ 1367.803374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1404.415384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1404.445812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.774647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1405.874676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1405.967426] Lustre: lustre-MDT0000: new disk, initializing [ 1406.117032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.134913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1410.927879] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1422.681021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1422.773982] Lustre: 44113:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1422.819588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1422.822508] Lustre: Skipped 1 previous similar message [ 1422.901914] Lustre: lustre-MDT0001: new disk, initializing [ 1423.036321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1423.068767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1423.085527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1427.477436] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1435.080207] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1439.046541] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1446.530729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1446.794965] Lustre: lustre-OST0000: new disk, initializing [ 1446.798751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1446.810842] Lustre: 45862:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1446.890870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1452.670450] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1454.716510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1454.733719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1454.856186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1461.389945] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1466.112196] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1466.349915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1470.662707] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1470.833994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1475.552193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1475.564836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.566786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1475.574935] Lustre: Skipped 1 previous similar message [ 1475.589422] Lustre: Skipped 1 previous similar message [ 1480.678326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1480.697737] Lustre: Skipped 1 previous similar message [ 1485.798756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1485.809649] Lustre: Skipped 1 previous similar message [ 1487.327151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1487.356106] LustreError: 47052:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1487.450159] Lustre: server umount lustre-OST0000 complete [ 1495.008342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1495.017077] LustreError: Skipped 1 previous similar message [ 1495.019446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.025615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.036771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.542459] LustreError: 47255:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1497.566585] LustreError: 47255:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1497.748873] Lustre: server umount lustre-MDT0000 complete [ 1501.356506] LustreError: 44889:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685739 with bad export cookie 12964173065615164648 [ 1501.359617] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.362942] LustreError: 44889:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1501.564704] LustreError: 47457:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1501.569789] LustreError: 47457:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1501.995683] Lustre: server umount lustre-MDT0001 complete [ 1511.570389] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:09:08 (1770685748) [ 1518.263412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.688594] LustreError: 48178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1518.709186] LustreError: 48178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1518.764831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.972230] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1524.194600] LustreError: 48178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.289567] LustreError: 48178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.984196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1531.466167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1536.355389] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1542.349188] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1546.233545] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1553.372135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1553.726795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.364807] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1565.985595] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1571.974247] LustreError: 50369:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1572.952807] Lustre: Modifying parameter general.timeout=1 in log params [ 1574.840554] Lustre: Modifying parameter general.timeout=3 in log params [ 1574.844323] Lustre: Skipped 1 previous similar message [ 1577.497764] Lustre: Modifying parameter general.timeout=6 in log params [ 1577.504297] Lustre: Skipped 2 previous similar messages [ 1581.805893] Lustre: Modifying parameter general.timeout=11 in log params [ 1581.814286] Lustre: Skipped 4 previous similar messages [ 1590.212886] Lustre: Modifying parameter general.timeout=22 in log params [ 1590.224373] Lustre: Skipped 10 previous similar messages [ 1606.320679] Lustre: Modifying parameter general.timeout=41 in log params [ 1606.323682] Lustre: Skipped 18 previous similar messages [ 1639.180290] Lustre: Modifying parameter general.timeout=76 in log params [ 1639.187095] Lustre: Skipped 34 previous similar messages [ 1699.026980] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:12:15 (1770685935) [ 1702.367640] Lustre: 57319:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1703.225471] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1703.234423] Lustre: Skipped 56 previous similar messages [ 1704.449268] Lustre: 57418:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1708.359978] Lustre: 57630:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1708.367569] Lustre: 57630:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1712.506544] Lustre: 57827:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1712.513308] Lustre: 57827:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1740.337754] Lustre: 59194:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1740.360745] LustreError: 59194:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1744.677471] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:13:01 (1770685981) [ 1756.641132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.645410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.654116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1756.654124] Lustre: Skipped 2 previous similar messages [ 1756.677735] Lustre: Skipped 2 previous similar messages [ 1762.803739] LustreError: 59895:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1762.811574] LustreError: 59895:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1762.893432] Lustre: server umount lustre-MDT0000 complete [ 1766.421420] LustreError: 48157:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686004 with bad export cookie 12964173065615165530 [ 1766.422217] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.432516] LustreError: 48157:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1766.884173] Lustre: server umount lustre-MDT0001 complete [ 1780.684409] LustreError: 60297:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1780.688619] LustreError: 60297:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1780.780266] Lustre: server umount lustre-OST0000 complete [ 1804.373647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1804.684304] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1804.901374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.310625] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1816.668342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1816.791780] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1816.833940] Lustre: 61723:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1821.034648] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1826.418732] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1830.038653] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1836.719232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1836.868139] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1837.095768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.098292] Lustre: Skipped 1 previous similar message [ 1842.748193] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1848.102264] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1849.630298] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1849.637067] Lustre: Skipped 20 previous similar messages [ 1849.671608] Lustre: 63950:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1855.975793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.979473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.013190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1856.026501] Lustre: Skipped 5 previous similar messages [ 1861.677153] LustreError: 64208:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1861.686463] LustreError: 64208:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1861.798214] Lustre: server umount lustre-OST0000 complete [ 1866.727586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.746131] Lustre: Skipped 2 previous similar messages [ 1871.717013] Lustre: server umount lustre-MDT0000 complete [ 1871.841599] LustreError: 61730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.855720] LustreError: 61730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1875.611111] LustreError: 61717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686113 with bad export cookie 12964173065615168589 [ 1875.612154] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.629933] LustreError: 61717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1876.032549] Lustre: server umount lustre-MDT0001 complete [ 1886.906895] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1889.694495] Key type lgssc unregistered [ 1889.937665] LNet: 65144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.950498] LNetError: 65144:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.968317] LNet: Removed LNI 192.168.204.159@tcp [ 1890.595244] Key type .llcrypt unregistered [ 1890.598813] Key type ._llcrypt unregistered [ 1915.789924] Key type ._llcrypt registered [ 1915.796218] Key type .llcrypt registered [ 1915.885445] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1917.026295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.056859] alg: No test for adler32 (adler32-zlib) [ 1918.128499] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1918.371617] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1920.031298] Key type lgssc registered [ 1920.906230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.223472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1928.240404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.858463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.741837] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1935.346638] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.365272] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1939.437198] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.731936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1942.150056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1945.935638] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1950.683862] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1954.784644] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1961.330261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1961.746564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.581880] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1974.044422] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1982.581593] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1993.184508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1993.188362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.201061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1995.746870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1995.748403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.786779] Lustre: Skipped 1 previous similar message [ 1996.960480] LustreError: 69268:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1997.038802] Lustre: server umount lustre-MDT0000 complete [ 2000.872359] LustreError: 66559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.887861] LustreError: 66559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2001.006598] LustreError: 66544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686238 with bad export cookie 15043316387005469859 [ 2001.011348] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.016602] LustreError: 66544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2001.151382] LustreError: 69472:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2001.158993] LustreError: 69472:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2001.354762] Lustre: server umount lustre-MDT0001 complete [ 2015.981975] LustreError: 69675:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2015.991465] LustreError: 69675:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2016.177109] Lustre: server umount lustre-OST0000 complete [ 2039.487028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.715583] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2039.735197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2039.973225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.842684] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2052.767549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2052.846083] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2052.878925] Lustre: 71101:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2053.336364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2057.215753] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2063.289220] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2067.473176] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2074.351493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2074.506502] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2074.906109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.387912] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2085.114685] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2086.923814] Lustre: 73336:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2088.480698] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2096.108171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2096.108754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.124364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2096.137211] Lustre: Skipped 1 previous similar message [ 2096.143018] Lustre: Skipped 2 previous similar messages [ 2098.350963] LustreError: 73585:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2098.352885] LustreError: 73585:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2098.456074] Lustre: server umount lustre-OST0000 complete [ 2104.287956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2104.298901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.312242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.315785] Lustre: Skipped 1 previous similar message [ 2107.560441] LustreError: 73792:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2107.563954] LustreError: 73792:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2107.623951] Lustre: server umount lustre-MDT0000 complete [ 2111.457045] LustreError: 71114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.472704] LustreError: 71114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2111.475706] LustreError: 71094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686349 with bad export cookie 15043316387005470699 [ 2111.487989] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.492077] LustreError: 71094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2111.724775] Lustre: server umount lustre-MDT0001 complete [ 2121.757445] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2124.148881] Key type lgssc unregistered [ 2124.461868] LNet: 74525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.466477] LNetError: 74525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.483394] LNet: Removed LNI 192.168.204.159@tcp [ 2125.095091] Key type .llcrypt unregistered [ 2125.098605] Key type ._llcrypt unregistered [ 2148.026070] Key type ._llcrypt registered [ 2148.027684] Key type .llcrypt registered [ 2148.134446] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2148.920049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.971198] alg: No test for adler32 (adler32-zlib) [ 2150.026974] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2150.305292] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2151.983206] Key type lgssc registered [ 2152.897603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2159.776993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2159.797829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.165154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.312459] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2170.344244] LustreError: 75919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.357984] LustreError: 75919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2173.574303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2174.058293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.611591] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2182.983983] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2186.962917] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2192.959501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2193.414922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.349949] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2204.009313] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2214.761436] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2215.564076] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2216.958971] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2216.964701] Lustre: Skipped 1 previous similar message [ 2219.220192] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2219.223349] Lustre: Skipped 2 previous similar messages [ 2223.405149] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2223.408868] Lustre: Skipped 5 previous similar messages [ 2231.737909] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2231.745125] Lustre: Skipped 11 previous similar messages [ 2247.859337] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2247.865430] Lustre: Skipped 19 previous similar messages [ 2280.330521] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2280.340430] Lustre: Skipped 42 previous similar messages [ 2296.295964] Lustre: 83670:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2298.870713] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:22:15 (1770686535) [ 2300.208199] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2301.624712] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:22:18 (1770686538) [ 2306.017296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.033255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.051068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2308.066768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.073488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2310.660108] LustreError: 83966:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2310.758504] Lustre: server umount lustre-OST0000 complete [ 2317.280187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2317.286213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.299139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.304871] Lustre: Skipped 1 previous similar message [ 2320.256591] LustreError: 84168:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2320.260592] LustreError: 84168:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2320.385215] Lustre: server umount lustre-MDT0000 complete [ 2323.427627] LustreError: 75919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.455900] LustreError: 75919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2323.759544] LustreError: 77544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686561 with bad export cookie 17654738920112112300 [ 2323.759980] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.771280] LustreError: 77544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2323.925567] LustreError: 84369:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2323.928834] LustreError: 84369:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2324.118014] Lustre: server umount lustre-MDT0001 complete [ 2333.569600] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2335.857708] Key type lgssc unregistered [ 2336.063285] LNet: 84901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.068853] LNetError: 84901:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.082651] LNet: Removed LNI 192.168.204.159@tcp [ 2336.674850] Key type .llcrypt unregistered [ 2336.676400] Key type ._llcrypt unregistered [ 2339.991508] Key type ._llcrypt registered [ 2339.996774] Key type .llcrypt registered [ 2340.078534] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2343.257269] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules [ 2344.200853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.213985] alg: No test for adler32 (adler32-zlib) [ 2345.193199] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2345.362612] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2346.991917] Key type lgssc registered [ 2347.827339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2348.800464] LustreError: 86066:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 2361.954606] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2368.857886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2371.057821] LustreError: 86066:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2371.101979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.651725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.210834] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2382.820173] LustreError: 87072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.832573] LustreError: 87072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2383.222786] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:23:40 (1770686620) [ 2386.866969] LustreError: 87103:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2386.874944] LustreError: 87103:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2386.886507] LustreError: 87103:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2387.010027] LustreError: 87876:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2389.096424] LustreError: 87876:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2389.102647] LustreError: 87876:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2389.195276] Lustre: server umount lustre-MDT0000 complete [ 2398.803636] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2400.667691] Key type lgssc unregistered [ 2400.878762] LNet: 88561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.883691] LNetError: 88561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.897709] LNet: Removed LNI 192.168.204.159@tcp [ 2402.628230] Key type .llcrypt unregistered [ 2402.633467] Key type ._llcrypt unregistered [ 2415.875804] Key type ._llcrypt registered [ 2415.877853] Key type .llcrypt registered [ 2415.952182] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2416.763294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2416.780411] alg: No test for adler32 (adler32-zlib) [ 2417.753173] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2417.906595] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2419.527409] Key type lgssc registered [ 2420.327512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2426.364549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2426.406536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.858709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.194843] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2438.119822] LustreError: 89737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.132035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2438.135362] LustreError: 89737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2438.634765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.892476] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2445.752711] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2449.259996] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2454.617168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2455.176539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.058361] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2464.374691] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2512.941140] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:25:49 (1770686749) [ 2514.268364] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2516.188469] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:25:52 (1770686752) [ 2519.011129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.013506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.018710] Lustre: Skipped 2 previous similar messages [ 2519.027874] Lustre: Skipped 2 previous similar messages [ 2524.132978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.140026] Lustre: Skipped 2 previous similar messages [ 2525.148376] LustreError: 92663:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2525.258115] Lustre: server umount lustre-MDT0000 complete [ 2528.273984] LustreError: 92665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686765 with bad export cookie 9489567439236845059 [ 2528.277363] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.280971] LustreError: 92665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2528.408085] LustreError: 92867:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2528.410125] LustreError: 92867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2528.582443] Lustre: server umount lustre-MDT0001 complete [ 2541.889626] LustreError: 93069:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2541.892390] LustreError: 93069:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2542.012538] Lustre: server umount lustre-OST0000 complete [ 2550.233813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.603447] LustreError: 93699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.746697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.067726] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2555.878352] LustreError: 93699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.410148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2560.982457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2563.402224] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2566.843527] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2569.074470] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2577.939248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2578.087378] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2578.092229] LustreError: 93688:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2578.112728] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2578.118163] LustreError: 95557:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -98 [ 2578.124372] LustreError: 95557:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2578.127936] LustreError: 95557:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2578.173692] Lustre: server umount lustre-OST0000 complete [ 2578.175749] LustreError: 95557:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2583.645169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2583.757612] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2583.763365] LustreError: 93686:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2583.769827] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2583.779764] LustreError: 96016:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -98 [ 2583.784700] LustreError: 96016:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2583.789854] LustreError: 96016:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2583.827476] Lustre: server umount lustre-OST0000 complete [ 2583.830203] LustreError: 96016:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2589.364988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2589.457950] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2589.462185] Lustre: Found index 0 for lustre-OST0000, updating log [ 2589.464530] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2589.514916] Lustre: lustre-OST0000: new disk, initializing [ 2589.517935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2589.524025] Lustre: 96550:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2589.670512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.429943] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2595.866190] Lustre: Failing over lustre-OST0000 [ 2595.894141] LustreError: 97095:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2595.897521] LustreError: 97095:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2595.961511] Lustre: server umount lustre-OST0000 complete [ 2601.957669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.957840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2601.964253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.970162] Lustre: Skipped 1 previous similar message [ 2604.367770] LustreError: 97295:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2604.372404] LustreError: 97295:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2604.472448] Lustre: server umount lustre-MDT0000 complete [ 2606.051463] LustreError: 93694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.061917] LustreError: 93694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2606.698616] LustreError: 96549:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686844 with bad export cookie 9489567439236847796 [ 2606.702711] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2606.708114] LustreError: 96549:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2606.877486] Lustre: server umount lustre-MDT0001 complete [ 2611.577711] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:27:28 (1770686848) [ 2620.493645] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2627.299864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.562348] LustreError: 98862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.647167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.966089] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2635.280240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2638.644675] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2640.630294] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2645.162470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2650.023689] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2655.906031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2655.996415] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2656.066196] Lustre: lustre-OST0001: new disk, initializing [ 2656.068893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2656.076879] Lustre: 101133:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2660.209803] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2662.943307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2662.954114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2663.019363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2665.754529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2670.210194] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2679.266233] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2679.269724] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.275616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2679.277770] Lustre: Skipped 1 previous similar message [ 2681.320419] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.326448] Lustre: Skipped 1 previous similar message [ 2684.109946] LustreError: 102099:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2684.113889] LustreError: 102099:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2684.172879] Lustre: server umount lustre-MDT0001 complete [ 2686.435655] LustreError: 98861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.445075] LustreError: 98861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2690.602063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2690.684676] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2690.687435] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2690.727438] Lustre: 98850:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2691.147626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2691.155523] Lustre: Skipped 3 previous similar messages [ 2692.783896] LustreError: 102793:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2692.790921] LustreError: 102793:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2692.797229] LustreError: 98853:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2692.853428] LustreError: 102807:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.204.159@tcp: cfg command failed: rc = -17 [ 2692.859188] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2692.859188] [ 2693.450683] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2696.168694] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2698.774503] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 20:28:56 (1770686936) [ 2709.329098] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2720.227374] Lustre: 104701:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2733.832120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2736.519466] Lustre: 105834:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2813.919808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2813.921528] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.922052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2813.922060] Lustre: Skipped 4 previous similar messages [ 2813.928295] Lustre: Skipped 2 previous similar messages [ 2816.153744] LustreError: 106291:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2816.156423] LustreError: 106291:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2816.194947] Lustre: server umount lustre-MDT0000 complete [ 2818.495690] LustreError: 100348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687056 with bad export cookie 9489567439236848727 [ 2818.496608] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.510722] LustreError: 100348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2818.732121] Lustre: server umount lustre-MDT0001 complete [ 2831.973360] Lustre: server umount lustre-OST0000 complete [ 2843.591603] Lustre: server umount lustre-OST0001 complete [ 2847.035261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2853.725910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2859.839250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2866.888677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2880.114691] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2886.356131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.367382] Lustre: lustre-MDT0000: reset Object Index mappings [ 2886.638655] LustreError: 109766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.650337] LustreError: 109766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2886.725182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2889.383676] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2895.554785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2895.577848] Lustre: lustre-MDT0001: reset Object Index mappings [ 2898.754649] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2900.997414] Lustre: 110879:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2905.282511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2910.043739] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2911.932221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 2916.116761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2917.974155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2921.011180] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2921.083389] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 2921.089349] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2926.584294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2929.211800] Lustre: 112727:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2970.995881] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:33:28 (1770687208) [ 2998.240199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.248029] Lustre: Skipped 2 previous similar messages [ 2998.249756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.258376] Lustre: Skipped 3 previous similar messages [ 3004.198105] LustreError: 113082:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3004.202231] LustreError: 113082:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3004.287530] Lustre: server umount lustre-MDT0000 complete [ 3006.475571] LustreError: 109748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687244 with bad export cookie 9489567439236895578 [ 3006.477594] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.483086] LustreError: 109748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3008.488782] LustreError: 109761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.498492] LustreError: 109761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3012.790962] Lustre: server umount lustre-MDT0001 complete [ 3021.399502] Lustre: server umount lustre-OST0000 complete [ 3029.727762] Lustre: server umount lustre-OST0001 complete [ 3033.004926] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 3037.806833] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3058.346421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.530916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3058.549675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3058.601494] Lustre: lustre-MDT0000: new disk, initializing [ 3058.661451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.665762] Lustre: Skipped 3 previous similar messages [ 3058.682454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3061.261669] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3068.071789] Lustre: server umount lustre-MDT0000 complete [ 3074.574722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.805418] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3074.816242] Lustre: 116938:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3074.828845] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3078.161057] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3082.920545] LustreError: 117452:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3082.923195] LustreError: 117452:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3083.076927] Lustre: server umount lustre-MDT0000 complete [ 3088.477899] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:35:25 (1770687325) [ 3089.356258] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3090.415749] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:35:27 (1770687327) [ 3099.535561] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3105.636168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.255366] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3113.819527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3113.916513] Lustre: 118951:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3113.957705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3113.960045] Lustre: Skipped 1 previous similar message [ 3114.015639] Lustre: lustre-MDT0001: new disk, initializing [ 3114.123413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3114.128289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3116.866620] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3120.450485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3124.585281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3124.761435] Lustre: lustre-OST0000: new disk, initializing [ 3124.765333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3124.776290] Lustre: 120495:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3126.349951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3126.357127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3126.406659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3128.009523] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3134.901226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3135.025458] Lustre: lustre-OST0001: new disk, initializing [ 3135.034087] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3135.041901] Lustre: 121348:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3137.080526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3137.089186] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3137.154143] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3138.874557] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3145.432731] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3147.857445] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3408.965381] Lustre: 127724:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3408.970256] Lustre: 127724:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3408.973092] Lustre: 127724:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3408.976313] Lustre: 127724:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3408.978984] Lustre: 127724:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3408.981661] Lustre: 127724:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3579.968588] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:43:37 (1770687817) [ 3592.677584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.681107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3592.684478] Lustre: Skipped 6 previous similar messages [ 3592.690163] Lustre: Skipped 12 previous similar messages [ 3598.159868] LustreError: 127943:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3598.162736] LustreError: 127943:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3598.243925] Lustre: server umount lustre-MDT0000 complete [ 3600.153981] LustreError: 118944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687837 with bad export cookie 9489567439236959873 [ 3600.155145] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.159211] LustreError: 118944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3602.912277] LustreError: 122159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.921050] LustreError: 122159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3606.452231] Lustre: server umount lustre-MDT0001 complete [ 3614.606714] Lustre: server umount lustre-OST0000 complete [ 3622.711561] Lustre: server umount lustre-OST0001 complete [ 3626.074982] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 3630.773909] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3647.858979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.012737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3648.028747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3648.069981] Lustre: lustre-MDT0000: new disk, initializing [ 3648.108880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3648.111979] Lustre: Skipped 5 previous similar messages [ 3648.123823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3650.282229] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3656.056074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3656.105872] Lustre: 130717:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3656.231278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3658.219750] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3662.501102] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3664.174420] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3667.185277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3667.311471] Lustre: 132456:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3668.487332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3668.491606] Lustre: Skipped 1 previous similar message [ 3668.494067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3668.550800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3670.135408] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3674.263349] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3679.382586] Lustre: lustre-MDD0000: changelog on [ 3681.000943] Lustre: lustre-MDD0001: changelog on [ 4378.349078] Lustre: lustre-MDD0001: changelog off [ 4379.222091] Lustre: 130723:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4379.658516] Lustre: lustre-MDD0000: changelog off [ 4384.736098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4384.744341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.751100] Lustre: Skipped 2 previous similar messages [ 4384.760492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.763596] Lustre: Skipped 9 previous similar messages [ 4385.248881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4388.210960] LustreError: 140223:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4388.212964] LustreError: 140223:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4388.270342] Lustre: server umount lustre-OST0000 complete [ 4390.369333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.372610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4390.374214] Lustre: Skipped 2 previous similar messages [ 4390.385367] Lustre: Skipped 3 previous similar messages [ 4400.608322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.613912] Lustre: Skipped 3 previous similar messages [ 4404.191157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4404.326872] Lustre: server umount lustre-MDT0000 complete [ 4405.728447] LustreError: 131691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4405.733946] LustreError: 131691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4406.052239] LustreError: 130708:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688643 with bad export cookie 9489567439237768184 [ 4406.054382] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4406.056699] LustreError: 130708:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4406.235396] Lustre: server umount lustre-MDT0001 complete [ 4410.419581] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 4411.662211] Key type lgssc unregistered [ 4411.805717] LNet: 141158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4411.809300] LNetError: 141158:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4411.819634] LNet: Removed LNI 192.168.204.159@tcp [ 4412.181208] Key type .llcrypt unregistered [ 4412.182997] Key type ._llcrypt unregistered [ 4420.267729] Key type ._llcrypt registered [ 4420.268974] Key type .llcrypt registered [ 4420.310082] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4426.861977] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4427.154733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4427.161591] alg: No test for adler32 (adler32-zlib) [ 4428.051447] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4428.138738] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 4429.727175] Key type lgssc registered [ 4430.104753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4445.048064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4445.062660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.198120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4446.217590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4446.267391] Lustre: lustre-MDT0000: new disk, initializing [ 4446.307350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.319034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4448.559073] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4454.056694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4454.123325] Lustre: 144296:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4454.149988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4454.151927] Lustre: Skipped 1 previous similar message [ 4454.192741] Lustre: lustre-MDT0001: new disk, initializing [ 4454.230539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4454.245313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4454.252907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4456.101953] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4459.885495] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4461.690473] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4464.928433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4465.091977] Lustre: lustre-OST0000: new disk, initializing [ 4465.098430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4465.099394] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4465.103906] Lustre: 146035:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4465.158681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4467.951871] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4470.274528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4470.280794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4470.297629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4471.806402] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4474.700129] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_module obdecho/obdecho [ 4475.639641] LustreError: 146905:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4475.643254] LustreError: 146905:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4476.435404] LustreError: 147002:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4480.999949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.002435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4481.004854] Lustre: Skipped 2 previous similar messages [ 4481.011855] Lustre: Skipped 2 previous similar messages [ 4486.114850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4486.118186] Lustre: Skipped 2 previous similar messages [ 4486.359595] LustreError: 147343:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4486.414358] Lustre: server umount lustre-MDT0000 complete [ 4487.981251] LustreError: 144287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688725 with bad export cookie 6687000278889427074 [ 4487.984329] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4487.986209] LustreError: 144287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4488.066541] LustreError: 147545:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4488.070366] LustreError: 147545:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4488.139721] Lustre: server umount lustre-MDT0001 complete [ 4500.018660] LustreError: 147745:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4500.023409] LustreError: 147745:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4500.091335] Lustre: server umount lustre-OST0000 complete [ 4508.523420] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4512.201674] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4527.326812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.488503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4527.513325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4527.568452] Lustre: lustre-MDT0000: new disk, initializing [ 4527.607138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.618851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4529.326351] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4534.106917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4534.153042] Lustre: 150677:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4534.188369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4534.190590] Lustre: Skipped 1 previous similar message [ 4534.228019] Lustre: lustre-MDT0001: new disk, initializing [ 4534.265723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4534.286693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4534.290991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4535.922437] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4539.353123] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4541.086728] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4543.897310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4544.037574] Lustre: lustre-OST0000: new disk, initializing [ 4544.039927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4544.043460] Lustre: 152417:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4544.090188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4546.347841] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4547.006920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4547.012202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4547.051375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4549.688317] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4551.363553] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4551.460691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4553.005646] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4553.087624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4557.280417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4557.282872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.286948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4557.288753] Lustre: Skipped 1 previous similar message [ 4559.927241] LustreError: 153609:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4559.931032] LustreError: 153609:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4559.996104] Lustre: server umount lustre-OST0000 complete [ 4562.401843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.403135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4562.408482] Lustre: Skipped 2 previous similar messages [ 4562.412416] Lustre: Skipped 1 previous similar message [ 4567.615153] LustreError: 153810:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4567.617937] LustreError: 153810:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4567.649812] Lustre: server umount lustre-MDT0000 complete [ 4569.038241] LustreError: 153811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688806 with bad export cookie 6687000278889428040 [ 4569.041975] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4569.042707] LustreError: 153811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4569.171127] Lustre: server umount lustre-MDT0001 complete [ 4570.548477] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:00:08 (1770688808) [ 4571.202033] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4571.930711] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:00:09 (1770688809) [ 4574.922790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.150950] LustreError: 154682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.158549] LustreError: 154682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4575.189809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.706441] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4580.120442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4581.924761] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4584.149972] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4585.689336] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4588.583740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4591.167891] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4593.518338] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4609.313492] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 4609.317334] Lustre: Skipped 2 previous similar messages [ 4610.528413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.533279] Lustre: Skipped 2 previous similar messages [ 4613.142909] LustreError: 156947:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4613.145975] LustreError: 156947:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4613.209578] Lustre: server umount lustre-MDT0000 complete [ 4614.423626] LustreError: 154678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.885847] LustreError: 154662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688852 with bad export cookie 6687000278889428922 [ 4614.890644] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4614.890935] LustreError: 154662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4615.648048] LustreError: 156293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.648196] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4619.542402] LustreError: 154678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.054187] Lustre: server umount lustre-MDT0001 complete [ 4623.462697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4628.308673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4628.414110] LustreError: 158218:0:(llog.c:1643:llog_backup()) MGC192.168.204.159@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4628.417900] Lustre: 158218:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.159@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4638.999609] LustreError: 158224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4639.006831] LustreError: 158224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4649.238199] LustreError: 158224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.244469] LustreError: 158224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4666.648953] LustreError: 158223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.658122] LustreError: 158223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4689.887340] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4689.889716] Lustre: 158218:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4700.128791] LustreError: 158242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.136218] LustreError: 158242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 4716.511304] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4716.513557] Lustre: 154673:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4730.344893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4730.349297] Lustre: Skipped 2 previous similar messages [ 4731.988939] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4734.436097] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4735.291242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.543425] LustreError: 154673:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4760.034528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 4760.038201] LustreError: Skipped 1 previous similar message [ 4760.040594] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5cccff530c249121 [ 4760.043685] Lustre: MGC192.168.204.159@tcp: Connection restored to 0@lo (at 0@lo) [ 4761.671064] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4763.619685] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4764.221266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4765.158121] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4765.490784] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4768.133111] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4774.664692] LustreError: 159700:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4774.667323] LustreError: 159700:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4774.696813] Lustre: server umount lustre-MDT0000 complete [ 4775.392752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4775.394160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.394818] LustreError: 158242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4775.394825] LustreError: 158242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 45 previous similar messages [ 4775.408400] Lustre: Skipped 2 previous similar messages [ 4775.946180] LustreError: 157150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689013 with bad export cookie 6687000278889435425 [ 4775.948827] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4775.950239] LustreError: 157150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4776.075100] Lustre: server umount lustre-MDT0001 complete [ 4787.680923] Lustre: server umount lustre-OST0000 complete [ 4794.876455] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4798.698762] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4811.651733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4811.737238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4811.749952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4811.789365] Lustre: lustre-MDT0000: new disk, initializing [ 4811.814054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.816739] Lustre: Skipped 1 previous similar message [ 4811.822948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4813.173947] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4817.320177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4817.358979] Lustre: 163030:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4817.463498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4818.774257] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4821.907967] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4823.158943] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4825.442306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4825.534630] Lustre: 164773:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4825.707742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4825.745208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4827.449323] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4830.777719] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4832.153934] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4832.225714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4833.572887] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4833.646119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4836.322161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4836.322931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.329814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.331711] Lustre: Skipped 7 previous similar messages [ 4840.429025] LustreError: 165964:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4840.430814] LustreError: 165964:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4840.469352] Lustre: server umount lustre-OST0000 complete [ 4842.975753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4842.978197] LustreError: Skipped 1 previous similar message [ 4847.845174] Lustre: server umount lustre-MDT0000 complete [ 4849.187346] LustreError: 163022:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689086 with bad export cookie 6687000278889454010 [ 4849.190995] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4849.191490] LustreError: 163022:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4849.321310] Lustre: server umount lustre-MDT0001 complete [ 4852.202827] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:04:49 (1770689089) [ 4854.396461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.959939] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4858.916699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4860.537423] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4862.454861] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4863.737659] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4866.240100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4868.370394] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4870.269450] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4889.948253] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:05:27 (1770689127) [ 4891.104611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4891.107504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.112181] Lustre: Skipped 3 previous similar messages [ 4891.115767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4891.117896] Lustre: Skipped 4 previous similar messages [ 4897.171898] Lustre: server umount lustre-OST0000 complete [ 4904.574346] LustreError: 169694:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4904.576381] LustreError: 169694:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4904.604885] Lustre: server umount lustre-MDT0000 complete [ 4905.820243] LustreError: 167017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689143 with bad export cookie 6687000278889454878 [ 4905.822556] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4905.825846] LustreError: 167017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4905.946165] Lustre: server umount lustre-MDT0001 complete [ 4909.569150] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 4910.539952] Key type lgssc unregistered [ 4910.666356] LNet: 170427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4910.669981] LNetError: 170427:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4910.679287] LNet: Removed LNI 192.168.204.159@tcp [ 4910.948123] Key type .llcrypt unregistered [ 4910.949251] Key type ._llcrypt unregistered [ 4918.996960] Key type ._llcrypt registered [ 4918.998149] Key type .llcrypt registered [ 4919.036157] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4919.436456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4919.443272] alg: No test for adler32 (adler32-zlib) [ 4920.292931] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4920.373640] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 4921.959125] Key type lgssc registered [ 4922.337109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4925.078563] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4925.084460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4940.511297] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4945.631139] LustreError: 171626:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 4950.751522] LustreError: 171626:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4950.755880] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4950.757759] LustreError: MGC192.168.204.159@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4950.762076] LustreError: 171626:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4950.764937] LustreError: 171626:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -22 [ 4950.767817] LustreError: 171626:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4950.769990] LustreError: 171626:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4950.796298] Lustre: server umount lustre-OST0000 complete [ 4950.798472] LustreError: 171626:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4953.889800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.088093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.387571] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4959.367069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4959.510363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4960.866989] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4962.767703] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4964.069257] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4966.577655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4966.648914] LustreError: 173711:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4966.653216] Lustre: 173711:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.204.159@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4966.699413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4968.692454] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4970.634371] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4975.073955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4975.074018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4975.075195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4975.078181] Lustre: Skipped 1 previous similar message [ 4977.473955] LustreError: 174357:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4977.479331] LustreError: 174357:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4977.506950] Lustre: server umount lustre-MDT0000 complete [ 4978.883437] LustreError: 172107:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689216 with bad export cookie 10793538401462613791 [ 4978.886756] LustreError: 172107:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4978.887142] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4978.951752] LustreError: 174562:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4978.953457] LustreError: 174562:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4979.010123] Lustre: server umount lustre-MDT0001 complete [ 4990.575497] LustreError: 174762:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4990.577879] LustreError: 174762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4990.620544] Lustre: server umount lustre-OST0000 complete [ 4993.670471] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4996.870637] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5008.449827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.537209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5008.547842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5008.588710] Lustre: lustre-MDT0000: new disk, initializing [ 5008.612652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.619719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5009.898625] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5014.095553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5014.137994] Lustre: 177084:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5014.162976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5014.165109] Lustre: Skipped 1 previous similar message [ 5014.201646] Lustre: lustre-MDT0001: new disk, initializing [ 5014.230549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5014.246043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5014.251651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5015.591714] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5018.714282] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5019.965875] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5022.273399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5022.377929] Lustre: lustre-OST0000: new disk, initializing [ 5022.380027] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5022.383495] Lustre: 178826:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5022.410835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.607200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5023.610086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5023.641264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5024.322340] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5027.512823] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5028.858549] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5028.935357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5030.285810] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5030.364363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5033.953437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5033.953847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5033.956379] LustreError: Skipped 1 previous similar message [ 5033.956944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5033.960957] Lustre: Skipped 2 previous similar messages [ 5033.966745] Lustre: Skipped 3 previous similar messages [ 5037.101708] LustreError: 180015:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5037.103757] LustreError: 180015:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5037.138940] Lustre: server umount lustre-OST0000 complete [ 5039.072500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.073380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5039.077250] Lustre: Skipped 1 previous similar message [ 5044.192029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.194257] Lustre: Skipped 2 previous similar messages [ 5044.503684] Lustre: server umount lustre-MDT0000 complete [ 5045.663985] LustreError: 178825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689283 with bad export cookie 10793538401462614477 [ 5045.666893] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5045.667884] LustreError: 178825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5045.725732] LustreError: 180419:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5045.728191] LustreError: 180419:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5045.775711] Lustre: server umount lustre-MDT0001 complete [ 5048.523301] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:08:06 (1770689286) [ 5054.223206] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5055.163674] Key type lgssc unregistered [ 5055.287565] LNet: 181601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5055.289873] LNetError: 181601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5055.301387] LNet: Removed LNI 192.168.204.159@tcp [ 5055.588880] Key type .llcrypt unregistered [ 5055.589936] Key type ._llcrypt unregistered [ 5062.539472] Key type ._llcrypt registered [ 5062.540894] Key type .llcrypt registered [ 5062.579732] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5062.977045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5062.984567] alg: No test for adler32 (adler32-zlib) [ 5063.834669] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5063.919811] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5065.503148] Key type lgssc registered [ 5065.857052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5068.504152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5068.511555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.692570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.060483] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5073.997242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5074.126108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5075.496865] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5077.362424] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5078.607968] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5081.217669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.296753] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5081.298357] LustreError: 184409:0:(llog.c:1684:llog_backup()) MGC192.168.204.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5081.301676] Lustre: 184409:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.204.159@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5081.341925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.316485] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5085.166740] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5088.737385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.738868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5088.743081] Lustre: Skipped 2 previous similar messages [ 5088.746361] Lustre: Skipped 2 previous similar messages [ 5092.033428] LustreError: 185054:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5092.060057] Lustre: server umount lustre-MDT0000 complete [ 5093.343401] LustreError: 184436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689331 with bad export cookie 15601221738266586529 [ 5093.347338] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5093.348000] LustreError: 184436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5093.423736] LustreError: 185258:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5093.426037] LustreError: 185258:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5093.474968] Lustre: server umount lustre-MDT0001 complete [ 5105.074773] LustreError: 185461:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5105.077136] LustreError: 185461:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5105.116449] Lustre: server umount lustre-OST0000 complete [ 5108.213631] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 5111.443980] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5123.436780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.529621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5123.541188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5123.577564] Lustre: lustre-MDT0000: new disk, initializing [ 5123.602963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.610261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5124.945437] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5129.235203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5129.271192] Lustre: 187781:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5129.288182] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5129.289796] Lustre: Skipped 1 previous similar message [ 5129.291228] LustreError: 188546:0:(llog.c:1684:llog_backup()) MGC192.168.204.159@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5129.293496] LustreError: 188546:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5129.295219] Lustre: 188546:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.159@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5129.302071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5129.303740] Lustre: Skipped 1 previous similar message [ 5129.339855] Lustre: lustre-MDT0001: new disk, initializing [ 5129.374241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5129.385857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5129.389507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5130.743412] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5133.930424] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5135.142963] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5137.531616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5137.623401] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5137.624933] Lustre: Skipped 1 previous similar message [ 5137.626500] LustreError: 189499:0:(llog.c:1684:llog_backup()) MGC192.168.204.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5137.630292] LustreError: 189499:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5137.632824] Lustre: 189499:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.159@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5137.636565] Lustre: 189499:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5137.645629] Lustre: lustre-OST0000: new disk, initializing [ 5137.647939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5137.651232] Lustre: 189522:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5137.677241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5139.429209] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5140.407415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5140.410981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5140.451442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5142.488364] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5143.901809] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5143.996872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5145.409277] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5145.497848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5150.689969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.690082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.690819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.695771] Lustre: Skipped 1 previous similar message [ 5150.698183] LustreError: Skipped 1 previous similar message [ 5152.365091] LustreError: 190714:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5152.366810] LustreError: 190714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5152.402402] Lustre: server umount lustre-OST0000 complete [ 5155.296926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5155.300986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.307252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5155.309632] Lustre: Skipped 1 previous similar message [ 5159.805673] LustreError: 190915:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5159.808140] LustreError: 190915:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5159.836139] Lustre: server umount lustre-MDT0000 complete [ 5160.928063] LustreError: 187793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.934437] LustreError: 187793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5161.127529] LustreError: 187773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689398 with bad export cookie 15601221738266587215 [ 5161.129850] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.131362] LustreError: 187773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5161.261769] Lustre: server umount lustre-MDT0001 complete [ 5164.037902] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:10:01 (1770689401) [ 5168.793039] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5172.211487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.345146] LustreError: 192484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.361063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5173.652916] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5176.644300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5178.034664] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5178.993958] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5181.165738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5183.103202] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5185.822674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5185.887857] Lustre: lustre-OST0001: new disk, initializing [ 5185.890182] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5185.893925] Lustre: 194737:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5187.207646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5187.212680] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5187.247641] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5187.830113] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5193.146279] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5195.692115] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5201.331561] Lustre: DEBUG MARKER: ADD OST3 [ 5202.940457] loop: module loaded [ 5202.958519] loop0: detected capacity change from 0 to 204800000 [ 5203.039185] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5203.042404] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5203.046419] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5205.067504] loop0: detected capacity change from 0 to 204800000 [ 5206.953940] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5207.022590] Lustre: lustre-OST0003: new disk, initializing [ 5207.025559] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5207.029286] Lustre: 196458:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5207.066197] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5207.068491] Lustre: Skipped 3 previous similar messages [ 5207.976390] Lustre: DEBUG MARKER: STOP OST3 [ 5208.623259] Lustre: Failing over lustre-OST0003 [ 5208.645593] LustreError: 196843:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5208.647594] LustreError: 196843:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5208.666882] Lustre: server umount lustre-OST0003 complete [ 5209.044397] LustreError: 194121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.079374] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5213.001379] Lustre: DEBUG MARKER: START OST3 again [ 5213.474387] LustreError: 193936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.481049] LustreError: 193936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5215.239174] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5215.318506] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5216.684880] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5216.687320] Lustre: lustre-OST0003: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5216.690188] LustreError: lustre-OST0003-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 5216.696759] Lustre: Skipped 1 previous similar message [ 5216.999493] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5217.186178] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5220.330738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5220.334138] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5220.343430] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5230.706213] Lustre: Failing over lustre-OST0003 [ 5230.728208] LustreError: 198761:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 5230.730097] LustreError: 198761:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5230.749957] Lustre: server umount lustre-OST0003 complete [ 5235.686084] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5235.686087] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5235.686101] LustreError: Skipped 1 previous similar message [ 5235.687426] LustreError: 193935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.687432] LustreError: 193935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5235.688470] Lustre: Skipped 2 previous similar messages [ 5243.360797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5243.363911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5243.367906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5243.370573] Lustre: Skipped 1 previous similar message [ 5245.923818] LustreError: 194966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.928422] LustreError: 194966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5249.501856] Lustre: server umount lustre-MDT0000 complete [ 5250.714237] LustreError: 192466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689488 with bad export cookie 15601221738266588083 [ 5250.717241] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5250.718585] LustreError: 192466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5251.041933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5251.044380] Lustre: Skipped 5 previous similar messages [ 5260.256388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5260.258719] Lustre: Skipped 3 previous similar messages [ 5264.863120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5264.905916] LustreError: 199230:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 5264.907432] LustreError: 199230:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5264.969639] Lustre: server umount lustre-MDT0001 complete [ 5266.251587] Lustre: server umount lustre-OST0000 complete [ 5267.555396] Lustre: server umount lustre-OST0001 complete [ 5269.609220] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 5272.359574] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5284.862704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.958564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5284.968760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5285.002957] Lustre: lustre-MDT0000: new disk, initializing [ 5285.026969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5285.028807] Lustre: Skipped 1 previous similar message [ 5285.034147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5286.379457] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5290.659938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5290.699615] Lustre: 201803:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5290.799155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5292.112146] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5295.243332] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5296.390130] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5298.627075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5298.740798] Lustre: 203544:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5300.409331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5300.448682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5300.690678] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5303.787410] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5305.180082] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5305.267681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5306.666732] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5306.747477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5310.945743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5310.945824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.947064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5310.947070] Lustre: Skipped 2 previous similar messages [ 5310.948971] LustreError: Skipped 1 previous similar message [ 5310.952802] Lustre: Skipped 6 previous similar messages [ 5313.550664] Lustre: server umount lustre-OST0000 complete [ 5329.375171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5329.418323] LustreError: 204932:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5329.420309] LustreError: 204932:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 5329.452882] Lustre: server umount lustre-MDT0000 complete [ 5330.400395] LustreError: 202597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.405987] LustreError: 202597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5330.809501] LustreError: 201794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689568 with bad export cookie 15601221738266590330 [ 5330.812289] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5330.813673] LustreError: 201794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5330.942432] Lustre: server umount lustre-MDT0001 complete [ 5332.254066] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:12:49 (1770689569) [ 5337.756137] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 5340.989164] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5352.978846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5353.082142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5353.095046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5353.097516] Lustre: Skipped 3 previous similar messages [ 5353.137541] Lustre: lustre-MDT0000: new disk, initializing [ 5353.138751] Lustre: Skipped 2 previous similar messages [ 5353.169651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5353.171956] Lustre: Skipped 2 previous similar messages [ 5354.498315] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5358.736695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5358.776949] Lustre: 207963:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5358.880630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5360.180333] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5363.189645] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5364.415944] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5366.817849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5366.944809] Lustre: 209705:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5368.220905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5368.860130] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5371.998912] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5373.409104] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5373.487626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5374.841757] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5374.926459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5378.528530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5378.530097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.531497] LustreError: Skipped 1 previous similar message [ 5378.532184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5378.537724] Lustre: Skipped 3 previous similar messages [ 5378.539141] Lustre: Skipped 7 previous similar messages [ 5381.708996] Lustre: server umount lustre-OST0000 complete [ 5389.078119] Lustre: server umount lustre-MDT0000 complete [ 5390.272529] LustreError: 207954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689627 with bad export cookie 15601221738266591212 [ 5390.275343] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5390.277328] LustreError: 207954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5390.402916] Lustre: server umount lustre-MDT0001 complete [ 5393.072773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.224664] LustreError: 211771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.480394] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5397.288381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5398.632062] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5400.276575] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5401.401288] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5403.482818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5405.291708] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5406.874759] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5433.315050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.318960] Lustre: Skipped 3 previous similar messages [ 5433.320750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5434.639064] Lustre: server umount lustre-OST0000 complete [ 5443.551965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5443.553544] Lustre: Skipped 10 previous similar messages [ 5450.207085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5450.271255] Lustre: server umount lustre-MDT0000 complete [ 5451.472428] LustreError: 211795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689689 with bad export cookie 15601221738266592094 [ 5451.476715] LustreError: 211795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5451.477167] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5451.587263] Lustre: server umount lustre-MDT0001 complete [ 5540.303338] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5541.384234] Key type lgssc unregistered [ 5541.533368] LNet: 215095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5541.537467] LNetError: 215095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5541.544436] LNet: Removed LNI 192.168.204.159@tcp [ 5541.912125] Key type .llcrypt unregistered [ 5541.913897] Key type ._llcrypt unregistered [ 5550.927827] Key type ._llcrypt registered [ 5550.929761] Key type .llcrypt registered [ 5550.969518] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 5557.275777] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5557.712613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5557.722839] alg: No test for adler32 (adler32-zlib) [ 5558.573229] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5558.653049] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5560.231174] Key type lgssc registered [ 5560.615261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5572.165713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5572.173228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.260668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5573.272685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5573.308089] Lustre: lustre-MDT0000: new disk, initializing [ 5573.330266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.335755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5574.620711] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5578.931092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5578.989556] Lustre: 218385:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5579.024468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5579.026762] Lustre: Skipped 1 previous similar message [ 5579.062971] Lustre: lustre-MDT0001: new disk, initializing [ 5579.100721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5579.114348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5579.118319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5580.424296] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5583.516180] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5584.721974] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5586.965864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5587.075903] Lustre: lustre-OST0000: new disk, initializing [ 5587.078394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5587.082781] Lustre: 220125:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5587.120880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.471525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5588.475037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5588.508907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5588.889590] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5591.890901] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5593.164979] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5593.232813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5594.445518] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5594.517119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5598.690259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5598.690429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.691126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5598.700288] Lustre: Skipped 1 previous similar message [ 5601.200930] LustreError: 221315:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5601.235442] Lustre: server umount lustre-OST0000 complete [ 5603.808619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.814343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.816290] Lustre: Skipped 2 previous similar messages [ 5608.574520] LustreError: 221516:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5608.577011] LustreError: 221516:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5608.604639] Lustre: server umount lustre-MDT0000 complete [ 5609.736672] LustreError: 218376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689847 with bad export cookie 2157704719090937678 [ 5609.739734] LustreError: 218376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5609.740144] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5609.798646] LustreError: 221719:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5609.800670] LustreError: 221719:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5609.845529] Lustre: server umount lustre-MDT0001 complete [ 5615.382976] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5618.584488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.746556] LustreError: 222890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.753020] LustreError: 222890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5618.761812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5620.117627] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5622.760871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5622.865975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5624.066274] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5624.967451] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5626.993507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5628.791137] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5631.370076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5631.437264] Lustre: lustre-OST0001: new disk, initializing [ 5631.438832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5631.440917] Lustre: 225143:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5631.474453] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5631.477067] Lustre: Skipped 1 previous similar message [ 5631.625966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5631.630398] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5631.660913] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5633.188459] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5638.548240] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5642.154365] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5650.959728] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:18:08 (1770689888) [ 5653.473349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5653.474361] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.475951] LustreError: Skipped 1 previous similar message [ 5653.476924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5653.479032] Lustre: Skipped 3 previous similar messages [ 5657.729942] LustreError: 226255:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5657.732113] LustreError: 226255:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5657.769153] Lustre: server umount lustre-MDT0000 complete [ 5658.594528] LustreError: 224361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.598700] LustreError: 224361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5658.900029] LustreError: 223989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689896 with bad export cookie 2157704719090938546 [ 5658.903709] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5658.905377] LustreError: 223989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5659.003219] Lustre: server umount lustre-MDT0001 complete [ 5669.873188] LustreError: 226660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5669.874949] LustreError: 226660:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5669.889964] Lustre: server umount lustre-OST0000 complete [ 5681.201398] LustreError: 226861:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5681.203846] LustreError: 226861:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5681.241695] Lustre: server umount lustre-OST0001 complete [ 5683.101312] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 5685.899620] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5697.077580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.157273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5697.167513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5697.202928] Lustre: lustre-MDT0000: new disk, initializing [ 5697.226157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.233381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5698.573753] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5702.789574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5702.827941] Lustre: 229033:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5702.924408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5704.236763] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5707.324320] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5708.525190] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5710.736393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5710.828574] Lustre: lustre-OST0000: new disk, initializing [ 5710.829785] Lustre: Skipped 1 previous similar message [ 5710.831591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5710.833281] Lustre: Skipped 2 previous similar messages [ 5710.835258] Lustre: 230773:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5712.380043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5712.384383] Lustre: Skipped 1 previous similar message [ 5712.386067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5712.421295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5712.698994] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5715.845753] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5717.225988] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5717.300972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5718.638888] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5718.707477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5722.594307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.594348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5722.598650] Lustre: Skipped 1 previous similar message [ 5722.601751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5722.604556] Lustre: Skipped 4 previous similar messages [ 5725.422601] LustreError: 231960:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5725.425304] LustreError: 231960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5725.469415] Lustre: server umount lustre-OST0000 complete [ 5727.712720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.713260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.719113] Lustre: Skipped 1 previous similar message [ 5737.953361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5737.955764] Lustre: Skipped 4 previous similar messages [ 5741.023106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5741.082499] Lustre: server umount lustre-MDT0000 complete [ 5742.254037] LustreError: 229025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689979 with bad export cookie 2157704719090940646 [ 5742.256406] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5742.256805] LustreError: 229025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5742.365767] Lustre: server umount lustre-MDT0001 complete [ 5745.072883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.250256] LustreError: 232838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.274204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5745.276508] Lustre: Skipped 2 previous similar messages [ 5746.615925] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5749.557292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5751.055683] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5752.865696] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5754.048487] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5756.213108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5758.118733] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5759.807783] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5902.821199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.821987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5902.822624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5902.824490] Lustre: Skipped 1 previous similar message [ 5902.830074] LustreError: Skipped 2 previous similar messages [ 5916.127150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5916.148138] LustreError: 235118:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5916.150569] LustreError: 235118:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5916.192795] Lustre: server umount lustre-OST0000 complete [ 5923.539273] Lustre: server umount lustre-MDT0000 complete [ 5924.792214] LustreError: 232864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690162 with bad export cookie 2157704719090941528 [ 5924.795290] LustreError: 232864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5924.795658] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.898505] Lustre: server umount lustre-MDT0001 complete [ 6023.719462] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6024.761350] Key type lgssc unregistered [ 6024.895435] LNet: 236090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6024.898451] LNetError: 236090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6024.906300] LNet: Removed LNI 192.168.204.159@tcp [ 6025.199125] Key type .llcrypt unregistered [ 6025.200216] Key type ._llcrypt unregistered [ 6033.389209] Key type ._llcrypt registered [ 6033.390222] Key type .llcrypt registered [ 6033.428737] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6039.805683] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6040.208229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6040.277118] alg: No test for adler32 (adler32-zlib) [ 6041.137534] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6041.219195] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6042.799144] Key type lgssc registered [ 6043.209183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6057.719884] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6061.247740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.255255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.350189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6062.361893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6062.402984] Lustre: lustre-MDT0000: new disk, initializing [ 6062.427911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.435309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6063.799346] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6068.023088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6068.065830] Lustre: 240078:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6068.088439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6068.090331] Lustre: Skipped 1 previous similar message [ 6068.127545] Lustre: lustre-MDT0001: new disk, initializing [ 6068.162811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.176129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6068.181289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6069.620491] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6071.965780] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6074.328722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6074.451757] Lustre: lustre-OST0000: new disk, initializing [ 6074.455054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6074.459610] Lustre: 241671:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6074.499878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.467263] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6076.870695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6076.873673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6076.923870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6080.821535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6080.889604] Lustre: lustre-OST0001: new disk, initializing [ 6080.891687] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6080.895976] Lustre: 242528:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6080.934381] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6082.844790] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6087.167485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6087.171644] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6087.205517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6088.277535] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6093.992770] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6102.498721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6102.500579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6102.502427] Lustre: Skipped 1 previous similar message [ 6107.617733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6107.620510] Lustre: Skipped 6 previous similar messages [ 6108.104982] LustreError: 243590:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6108.136544] Lustre: server umount lustre-MDT0000 complete [ 6109.371369] LustreError: 240070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690347 with bad export cookie 4060543487318530966 [ 6109.374369] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6109.374676] LustreError: 240070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6109.448678] LustreError: 243793:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6109.452381] LustreError: 243793:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6109.512289] Lustre: server umount lustre-MDT0001 complete [ 6120.950380] LustreError: 243993:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6120.953097] LustreError: 243993:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6120.976110] Lustre: server umount lustre-OST0000 complete [ 6131.443227] LustreError: 244195:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6131.445016] LustreError: 244195:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6131.490142] Lustre: server umount lustre-OST0001 complete [ 6136.160622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6139.356882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6142.179670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6145.222610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6151.393687] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6154.963772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.970566] Lustre: lustre-MDT0000: reset Object Index mappings [ 6155.147237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6156.467060] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6159.343851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6159.348933] Lustre: lustre-MDT0001: reset Object Index mappings [ 6160.803745] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6161.736843] Lustre: 248604:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6163.856343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6163.983587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.985443] Lustre: Skipped 1 previous similar message [ 6165.028443] LustreError: 248958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6165.036587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6165.037023] LustreError: 248958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6165.842245] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6168.680510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6169.769875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6170.718840] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6173.826391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6174.978487] Lustre: 250443:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6185.443934] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.447518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6185.448654] Lustre: Skipped 4 previous similar messages [ 6185.449327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6187.201084] LustreError: 250547:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6187.203023] LustreError: 250547:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6187.235501] Lustre: server umount lustre-MDT0000 complete [ 6188.743897] LustreError: 250445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690426 with bad export cookie 4060543487318537742 [ 6188.748192] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.748676] LustreError: 250445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6188.900789] Lustre: server umount lustre-MDT0001 complete [ 6201.419133] LustreError: 250950:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6201.421941] LustreError: 250950:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6201.502532] Lustre: server umount lustre-OST0000 complete [ 6214.501864] Lustre: server umount lustre-OST0001 complete [ 6230.412922] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:27:47 (1770690467) [ 6236.744217] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6240.386137] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6254.021320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.145034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6254.162091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6254.205945] Lustre: lustre-MDT0000: new disk, initializing [ 6254.236745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6254.239278] Lustre: Skipped 1 previous similar message [ 6254.249491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6255.840201] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6260.528153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6260.576263] Lustre: 254660:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6260.703215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6262.336117] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6266.003508] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6267.517176] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6270.629841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6270.780244] Lustre: lustre-OST0000: new disk, initializing [ 6270.782670] Lustre: Skipped 1 previous similar message [ 6270.790794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6270.796980] Lustre: Skipped 2 previous similar messages [ 6270.803327] Lustre: 256402:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6271.971109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6271.974351] Lustre: Skipped 1 previous similar message [ 6271.976764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6272.039482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6273.490260] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6277.291100] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6278.980480] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6279.086524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6280.824539] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6280.912941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6282.209352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6282.210504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.211923] LustreError: Skipped 1 previous similar message [ 6282.216288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6282.219266] Lustre: Skipped 4 previous similar messages [ 6287.330874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6287.726142] LustreError: 257589:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6287.729281] LustreError: 257589:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6287.799645] Lustre: server umount lustre-OST0000 complete [ 6291.424925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6291.427954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.433927] Lustre: Skipped 1 previous similar message [ 6295.394964] Lustre: server umount lustre-MDT0000 complete [ 6296.958542] LustreError: 254651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690534 with bad export cookie 4060543487318539835 [ 6296.962103] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6296.962501] LustreError: 254651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6297.107901] Lustre: server umount lustre-MDT0001 complete [ 6303.639351] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6307.375746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.522501] LustreError: 259165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6307.538089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.540781] Lustre: Skipped 2 previous similar messages [ 6308.866755] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6311.868933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6313.384363] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6314.517660] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6317.120071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6319.643176] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6322.908550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6322.976657] Lustre: lustre-OST0001: new disk, initializing [ 6322.979223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6322.983112] Lustre: 261425:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6325.012700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6325.018463] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6325.068054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6325.350935] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6330.260321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6331.867121] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6338.924758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6338.933403] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6342.623963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6342.627665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.632819] Lustre: Skipped 1 previous similar message [ 6342.635051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.637366] Lustre: Skipped 4 previous similar messages [ 6347.077701] LustreError: 262667:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6347.081684] LustreError: 262667:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6347.121803] Lustre: server umount lustre-MDT0000 complete [ 6348.688735] LustreError: 259144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690586 with bad export cookie 4060543487318540717 [ 6348.694295] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6348.694704] LustreError: 259144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6348.822686] Lustre: server umount lustre-MDT0001 complete [ 6361.040591] Lustre: server umount lustre-OST0000 complete [ 6372.979507] Lustre: server umount lustre-OST0001 complete [ 6379.978660] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6384.383603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.606315] LustreError: 264444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.650900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.653763] Lustre: Skipped 3 previous similar messages [ 6386.461236] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6389.729724] LustreError: 264444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6390.481848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6392.535744] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6393.944936] Lustre: 265547:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6396.911302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6399.497626] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6401.189794] LustreError: 265901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6403.517462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6405.992347] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6409.710267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6409.712573] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6414.734166] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6416.172820] Lustre: 267400:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6423.215583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6423.241569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6426.847608] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:31:04 (1770690664) [ 6440.418566] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.421138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6440.424148] Lustre: Skipped 5 previous similar messages [ 6440.427224] Lustre: Skipped 5 previous similar messages [ 6444.622956] LustreError: 268107:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6444.625971] LustreError: 268107:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6444.688488] Lustre: server umount lustre-MDT0000 complete [ 6445.539207] LustreError: 264438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.546027] LustreError: 264438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6446.405968] LustreError: 267403:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690684 with bad export cookie 4060543487318547087 [ 6446.410033] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6446.410552] LustreError: 267403:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6452.613378] Lustre: server umount lustre-MDT0001 complete [ 6460.487948] Lustre: server umount lustre-OST0000 complete [ 6468.350528] Lustre: server umount lustre-OST0001 complete [ 6471.082796] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6475.162615] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6495.071689] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6499.690521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.801900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6499.819607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6499.874592] Lustre: lustre-MDT0000: new disk, initializing [ 6499.932152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6501.785347] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6506.784966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6506.831384] Lustre: 271580:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6506.967226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6508.638823] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6514.126386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6514.265289] Lustre: 273178:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6514.309750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.312582] Lustre: Skipped 5 previous similar messages [ 6514.944726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6516.606292] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6521.419528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6522.704770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6524.007651] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6529.220953] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6541.477856] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 21:32:59 (1770690779) [ 6542.815868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6542.820285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.830779] Lustre: Skipped 3 previous similar messages [ 6542.833165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6542.835781] Lustre: Skipped 4 previous similar messages [ 6548.518645] Lustre: server umount lustre-MDT0000 complete [ 6550.018578] LustreError: 271571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690787 with bad export cookie 4060543487318555025 [ 6550.019725] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6550.023162] LustreError: 271571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6550.152420] Lustre: server umount lustre-MDT0001 complete [ 6561.797608] Lustre: server umount lustre-OST0000 complete [ 6573.682131] LustreError: 275784:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6573.684139] LustreError: 275784:0:(obd_class.h:479:obd_check_dev()) Skipped 45 previous similar messages [ 6573.742788] Lustre: server umount lustre-OST0001 complete [ 6581.379724] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6585.353601] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6600.043907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6600.139831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6600.143589] Lustre: Skipped 2 previous similar messages [ 6600.154300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6600.156028] Lustre: Skipped 4 previous similar messages [ 6600.192541] Lustre: lustre-MDT0000: new disk, initializing [ 6600.193885] Lustre: Skipped 3 previous similar messages [ 6600.230616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6600.235094] Lustre: Skipped 3 previous similar messages [ 6601.737538] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6606.418473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6606.466127] Lustre: 278612:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6606.588894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6606.592242] Lustre: Skipped 2 previous similar messages [ 6608.118066] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6611.583123] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6613.050518] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6615.658402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6615.773906] Lustre: 280354:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6615.777637] Lustre: 280354:0:(osd_compat.c:1354:osd_object_spec_find()) Skipped 1 previous similar message [ 6617.835785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6618.072402] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6621.497671] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6622.299409] Lustre: Mounted lustre-client [ 6622.687190] LustreError: 281106:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9d322a924000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 6622.717221] Lustre: Unmounted lustre-client [ 6627.299809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.300726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6627.305263] Lustre: Skipped 4 previous similar messages [ 6627.307955] Lustre: Skipped 8 previous similar messages [ 6628.247478] Lustre: server umount lustre-OST0000 complete [ 6632.417045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6644.191119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6644.257830] Lustre: server umount lustre-MDT0000 complete [ 6645.741407] LustreError: 279390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690883 with bad export cookie 4060543487318556551 [ 6645.744124] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6645.746508] LustreError: 279390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6645.885271] Lustre: server umount lustre-MDT0001 complete [ 6649.928709] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6651.139166] Key type lgssc unregistered [ 6684.063366] LNet: 282386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6684.066561] LNetError: 282386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6751.654413] LNet: Removed LNI 192.168.204.159@tcp [ 6752.007124] Key type .llcrypt unregistered [ 6752.008918] Key type ._llcrypt unregistered [ 6758.554657] Key type ._llcrypt registered [ 6758.556020] Key type .llcrypt registered [ 6758.596772] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6765.178637] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6765.498350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6765.570335] alg: No test for adler32 (adler32-zlib) [ 6766.430815] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6766.515640] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6768.103149] Key type lgssc registered [ 6768.511522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6779.395609] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:36:57 (1770691017) [ 6781.829885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6781.837425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.938702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6782.959985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6782.997509] Lustre: lustre-MDT0000: new disk, initializing [ 6783.024755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.034666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6784.444096] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6788.971218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6789.023257] Lustre: 285540:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6789.055476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6789.057610] Lustre: Skipped 1 previous similar message [ 6789.097344] Lustre: lustre-MDT0001: new disk, initializing [ 6789.139566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6789.158578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6789.164813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6790.609442] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6794.082554] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6795.440051] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6798.001430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6798.109519] Lustre: lustre-OST0000: new disk, initializing [ 6798.112148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6798.117304] Lustre: 287282:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6798.159534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6798.978248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6798.982277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6799.023176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6800.441305] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6803.965883] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6809.569065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6809.571579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.574356] LustreError: Skipped 1 previous similar message [ 6809.580250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6809.586993] Lustre: Skipped 1 previous similar message [ 6814.690586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6814.692831] Lustre: Skipped 1 previous similar message [ 6819.811146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6819.813427] Lustre: Skipped 1 previous similar message [ 6822.367123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6822.385076] LustreError: 287976:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6822.423526] Lustre: server umount lustre-OST0000 complete [ 6824.928940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.929857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6824.930245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6824.933894] Lustre: Skipped 2 previous similar messages [ 6824.941757] Lustre: Skipped 1 previous similar message [ 6830.049401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6830.052790] Lustre: Skipped 1 previous similar message [ 6838.241086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6838.280894] LustreError: 288178:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6838.283056] LustreError: 288178:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6838.311143] Lustre: server umount lustre-MDT0000 complete [ 6839.825313] LustreError: 285533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691077 with bad export cookie 5933055282465999900 [ 6839.829517] LustreError: 285533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6839.829830] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6839.896949] LustreError: 288381:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6839.899317] LustreError: 288381:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6839.956522] Lustre: server umount lustre-MDT0001 complete [ 6843.882098] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6845.133109] Key type lgssc unregistered [ 6845.286426] LNet: 288912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6845.289715] LNetError: 288912:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6845.296312] LNet: Removed LNI 192.168.204.159@tcp [ 6845.647258] Key type .llcrypt unregistered [ 6845.648466] Key type ._llcrypt unregistered [ 6882.504603] Key type ._llcrypt registered [ 6882.505954] Key type .llcrypt registered [ 6882.546720] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6882.644456] Key type .llcrypt unregistered [ 6882.645855] Key type ._llcrypt unregistered [ 6912.026432] Key type ._llcrypt registered [ 6912.028304] Key type .llcrypt registered [ 6912.073723] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6912.179228] Key type .llcrypt unregistered [ 6912.180396] Key type ._llcrypt unregistered [ 6923.606847] Key type ._llcrypt registered [ 6923.608233] Key type .llcrypt registered [ 6923.651624] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6923.755512] Key type .llcrypt unregistered [ 6923.756878] Key type ._llcrypt unregistered [ 6935.719348] Key type ._llcrypt registered [ 6935.720894] Key type .llcrypt registered [ 6935.761468] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6935.874404] Key type .llcrypt unregistered [ 6935.875741] Key type ._llcrypt unregistered [ 6964.836992] Key type ._llcrypt registered [ 6964.838177] Key type .llcrypt registered [ 6964.880121] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6964.991355] Key type .llcrypt unregistered [ 6964.992698] Key type ._llcrypt unregistered [ 6976.193950] Key type ._llcrypt registered [ 6976.195534] Key type .llcrypt registered [ 6976.232330] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6976.331242] Key type .llcrypt unregistered [ 6976.332661] Key type ._llcrypt unregistered [ 6987.209687] Key type ._llcrypt registered [ 6987.210820] Key type .llcrypt registered [ 6987.250761] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6987.350384] Key type .llcrypt unregistered [ 6987.351353] Key type ._llcrypt unregistered [ 6998.806153] Key type ._llcrypt registered [ 6998.807557] Key type .llcrypt registered [ 6998.847544] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6998.954501] Key type .llcrypt unregistered [ 6998.955816] Key type ._llcrypt unregistered [ 7011.152342] Key type ._llcrypt registered [ 7011.153579] Key type .llcrypt registered [ 7011.203069] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7011.332612] Key type .llcrypt unregistered [ 7011.334446] Key type ._llcrypt unregistered [ 7023.564718] Key type ._llcrypt registered [ 7023.565846] Key type .llcrypt registered [ 7023.607245] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7023.715407] Key type .llcrypt unregistered [ 7023.716850] Key type ._llcrypt unregistered [ 7045.327555] Key type ._llcrypt registered [ 7045.328743] Key type .llcrypt registered [ 7045.365593] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7045.459224] Key type .llcrypt unregistered [ 7045.460647] Key type ._llcrypt unregistered [ 7056.304965] Key type ._llcrypt registered [ 7056.305906] Key type .llcrypt registered [ 7056.344415] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7056.444336] Key type .llcrypt unregistered [ 7056.445398] Key type ._llcrypt unregistered [ 7067.569034] Key type ._llcrypt registered [ 7067.570152] Key type .llcrypt registered [ 7067.608494] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7067.709249] Key type .llcrypt unregistered [ 7067.711182] Key type ._llcrypt unregistered [ 7087.234517] Key type ._llcrypt registered [ 7087.235686] Key type .llcrypt registered [ 7087.279924] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 7093.667509] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7094.037306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7094.077857] alg: No test for adler32 (adler32-zlib) [ 7094.944933] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 7095.029090] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7096.615188] Key type lgssc registered [ 7097.001875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7107.605947] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6791 sec ========== 21:42:25 (1770691345) [ 7108.142431] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:42:25 (1770691345) === [ 7109.253887] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:42:26 (1770691346) === [ 7122.579016] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7123.648518] Key type lgssc unregistered [ 7123.775407] LNet: 310608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7123.781423] LNetError: 310608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7123.788581] LNet: Removed LNI 192.168.204.159@tcp [ 7124.086975] Key type .llcrypt unregistered [ 7124.088742] Key type ._llcrypt unregistered