[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456130884 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002227] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.005861] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009160] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011063] SELinux: Initializing. [ 0.012045] *** VALIDATE selinux *** [ 0.019283] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024437] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027163] *** VALIDATE tmpfs *** [ 0.029335] *** VALIDATE proc *** [ 0.030298] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035025] Spectre V2 : User space: Vulnerable [ 0.036005] Speculative Store Bypass: Vulnerable [ 0.039051] debug: unmapping init [mem 0xffffffffaca59000-0xffffffffaca60fff] [ 0.040567] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042215] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043012] ... version: 2 [ 0.044007] ... bit width: 48 [ 0.045006] ... generic registers: 4 [ 0.046005] ... value mask: 0000ffffffffffff [ 0.047006] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.049007] ... event mask: 000000070000000f [ 0.051114] rcu: Hierarchical SRCU implementation. [ 0.053232] smp: Bringing up secondary CPUs ... [ 0.054510] x86: Booting SMP configuration: [ 0.055019] .... node #0, CPUs: #1 #2 #3 [ 0.058733] smp: Brought up 1 node, 4 CPUs [ 0.060007] smpboot: Max logical packages: 1 [ 0.060826] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.184833] node 0 deferred pages initialised in 123ms [ 0.188212] devtmpfs: initialized [ 0.189274] x86/mm: Memory block size: 128MB [ 0.191847] gcov: version magic: 0x41383552 [ 0.193391] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.196082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.198350] pinctrl core: initialized pinctrl subsystem [ 0.201229] [ 0.201695] ************************************************************* [ 0.203009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.204007] ** ** [ 0.206008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.207007] ** ** [ 0.208008] ** This means that this kernel is built to expose internal ** [ 0.210009] ** IOMMU data structures, which may compromise security on ** [ 0.212011] ** your system. ** [ 0.214011] ** ** [ 0.216009] ** If you see this message and you are not debugging the ** [ 0.218010] ** kernel, report this immediately to your vendor! ** [ 0.219006] ** ** [ 0.220006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221006] ************************************************************* [ 0.224119] NET: Registered protocol family 16 [ 0.225364] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.227063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.230079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.234045] cpuidle: using governor menu [ 0.236000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.239536] PCI: Using configuration type 1 for base access [ 0.241154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.251121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.253011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.256149] cryptd: max_cpu_qlen set to 1000 [ 0.258229] ACPI: Added _OSI(Module Device) [ 0.259018] ACPI: Added _OSI(Processor Device) [ 0.261026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.262010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.265805] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.272514] ACPI: Interpreter enabled [ 0.274058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.275009] ACPI: Using IOAPIC for interrupt routing [ 0.277108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.280417] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.287000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.289038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.290010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.292075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.296141] acpiphp: Slot [2] registered [ 0.297145] acpiphp: Slot [3] registered [ 0.299128] acpiphp: Slot [4] registered [ 0.300140] acpiphp: Slot [5] registered [ 0.302181] acpiphp: Slot [6] registered [ 0.304145] acpiphp: Slot [7] registered [ 0.305103] acpiphp: Slot [8] registered [ 0.307105] acpiphp: Slot [9] registered [ 0.308122] acpiphp: Slot [10] registered [ 0.309069] acpiphp: Slot [11] registered [ 0.310059] acpiphp: Slot [12] registered [ 0.311083] acpiphp: Slot [13] registered [ 0.312063] acpiphp: Slot [14] registered [ 0.314058] acpiphp: Slot [15] registered [ 0.315000] acpiphp: Slot [16] registered [ 0.315000] acpiphp: Slot [17] registered [ 0.317059] acpiphp: Slot [18] registered [ 0.318070] acpiphp: Slot [19] registered [ 0.319089] acpiphp: Slot [20] registered [ 0.320083] acpiphp: Slot [21] registered [ 0.321059] acpiphp: Slot [22] registered [ 0.322090] acpiphp: Slot [23] registered [ 0.324082] acpiphp: Slot [24] registered [ 0.325072] acpiphp: Slot [25] registered [ 0.326074] acpiphp: Slot [26] registered [ 0.328066] acpiphp: Slot [27] registered [ 0.329000] acpiphp: Slot [28] registered [ 0.329000] acpiphp: Slot [29] registered [ 0.331069] acpiphp: Slot [30] registered [ 0.332063] acpiphp: Slot [31] registered [ 0.334060] PCI host bridge to bus 0000:00 [ 0.335013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.342014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344013] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.347016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.349173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353404] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357414] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363799] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.368064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376429] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377458] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381716] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384891] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.395000] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.399014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405508] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.423020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446014] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.457730] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.470021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.492052] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.508847] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.559723] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.576022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.598019] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.609944] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.625021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.632014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.651015] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.661166] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.671094] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.682029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.705018] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.718814] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.721436] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.723320] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.726282] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.733039] iommu: Default domain type: Passthrough [ 0.735400] SCSI subsystem initialized [ 0.736070] ACPI: bus type USB registered [ 0.737081] usbcore: registered new interface driver usbfs [ 0.738041] usbcore: registered new interface driver hub [ 0.740044] usbcore: registered new device driver usb [ 0.741147] pps_core: LinuxPPS API ver. 1 registered [ 0.742008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.744056] PTP clock support registered [ 0.748093] EDAC MC: Ver: 3.0.0 [ 0.749518] PCI: Using ACPI for IRQ routing [ 0.750000] NetLabel: Initializing [ 0.751010] NetLabel: domain hash size = 128 [ 0.752008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.754090] NetLabel: unlabeled traffic allowed by default [ 0.756131] vgaarb: loaded [ 0.758237] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.760012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.768000] clocksource: Switched to clocksource kvm-clock [ 0.884462] VFS: Disk quotas dquot_6.6.0 [ 0.886110] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.888833] *** VALIDATE ramfs *** [ 0.890282] *** VALIDATE hugetlbfs *** [ 0.891811] pnp: PnP ACPI init [ 0.894113] pnp: PnP ACPI: found 6 devices [ 0.908727] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.911545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.913353] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.915140] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.917191] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.919191] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.921513] NET: Registered protocol family 2 [ 0.923848] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.929285] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.933106] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.937684] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.941118] TCP: Hash tables configured (established 65536 bind 65536) [ 0.943620] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.946287] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948797] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951623] NET: Registered protocol family 1 [ 0.955136] RPC: Registered named UNIX socket transport module. [ 0.956913] RPC: Registered udp transport module. [ 0.958761] RPC: Registered tcp transport module. [ 0.960577] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.963034] NET: Registered protocol family 44 [ 0.964565] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.966735] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.968837] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.971176] PCI: CLS 0 bytes, default 64 [ 0.972853] Unpacking initramfs... [ 2.532588] debug: unmapping init [mem 0xffff907ebcc54000-0xffff907ebffbffff] [ 2.540387] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.542254] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.544797] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.054306] Initialise system trusted keyrings [ 3.056112] Key type blacklist registered [ 3.058459] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.066609] zbud: loaded [ 3.071168] *** VALIDATE nfs *** [ 3.072371] *** VALIDATE nfs4 *** [ 3.073765] pstore: using deflate compression [ 3.077169] Platform Keyring initialized [ 3.240500] NET: Registered protocol family 38 [ 3.242352] Key type asymmetric registered [ 3.243515] Asymmetric key parser 'x509' registered [ 3.245250] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.248533] io scheduler mq-deadline registered [ 3.249874] io scheduler kyber registered [ 3.251271] io scheduler bfq registered [ 3.254712] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.257532] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.259827] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.261997] ACPI: Power Button [PWRF] [ 3.367923] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.472675] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.679975] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.782912] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.996652] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.029814] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.060572] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.064204] Non-volatile memory driver v1.3 [ 4.065391] Linux agpgart interface v0.103 [ 4.091758] virtio_blk virtio1: [vda] 132808 512-byte logical blocks (68.0 MB/64.8 MiB) [ 4.094246] vda: detected capacity change from 0 to 67997696 [ 4.109195] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.111627] vdb: detected capacity change from 0 to 1073741824 [ 4.127090] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.129399] vdc: detected capacity change from 0 to 2621440000 [ 4.145683] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.147971] vdd: detected capacity change from 0 to 2621440000 [ 4.166797] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.170501] vde: detected capacity change from 0 to 4294967296 [ 4.187864] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.190142] vdf: detected capacity change from 0 to 4294967296 [ 4.199232] libphy: Fixed MDIO Bus: probed [ 4.210212] usbcore: registered new interface driver usbserial_generic [ 4.212092] usbserial: USB Serial support registered for generic [ 4.213846] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.217167] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.218630] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.220234] mousedev: PS/2 mouse device common for all mice [ 4.222195] rtc_cmos 00:05: RTC can wake from S4 [ 4.225105] rtc_cmos 00:05: registered as rtc0 [ 4.227235] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.229172] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.236316] intel_pstate: CPU model not supported [ 4.246020] hid: raw HID events driver (C) Jiri Kosina [ 4.254356] usbcore: registered new interface driver usbhid [ 4.256872] usbhid: USB HID core driver [ 4.260224] drop_monitor: Initializing network drop monitor service [ 4.262086] Initializing XFRM netlink socket [ 4.263585] NET: Registered protocol family 10 [ 4.264525] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.271688] Segment Routing with IPv6 [ 4.271720] NET: Registered protocol family 17 [ 4.274377] mpls_gso: MPLS GSO support [ 4.280742] RAS: Correctable Errors collector initialized. [ 4.282222] AVX version of gcm_enc/dec engaged. [ 4.283446] AES CTR mode by8 optimization enabled [ 4.285566] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.381945] sched_clock: Marking stable (4381883750, 0)->(5271182330, -889298580) [ 4.388108] registered taskstats version 1 [ 4.390683] Loading compiled-in X.509 certificates [ 4.394258] zswap: loaded using pool lzo/zbud [ 4.424809] Key type big_key registered [ 4.435332] Key type encrypted registered [ 4.436598] ima: No TPM chip found, activating TPM-bypass! [ 4.438147] ima: Allocated hash algorithm: sha1 [ 4.439904] ima: No architecture policies found [ 4.441437] evm: Initialising EVM extended attributes: [ 4.442953] evm: security.selinux [ 4.443973] evm: security.ima [ 4.444930] evm: security.capability [ 4.445984] evm: HMAC attrs: 0x1 [ 4.449235] rtc_cmos 00:05: setting system clock to 2025-08-19 20:54:30 UTC (1755636870) [ 4.457365] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 4.459856] debug: unmapping init [mem 0xffffffffac782000-0xffffffffaca58fff] [ 4.468657] Write protecting the kernel read-only data: 28672k [ 4.471829] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 4.475269] debug: unmapping init [mem 0xffffffffab714000-0xffffffffab7fffff] [ 4.508351] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.517745] systemd[1]: Detected virtualization kvm. [ 4.519378] systemd[1]: Detected architecture x86-64. [ 4.521061] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.550186] systemd[1]: No hostname configured. [ 4.551534] systemd[1]: Set hostname to . [ 4.553181] random: systemd: uninitialized urandom read (16 bytes read) [ 4.555090] systemd[1]: Initializing machine ID from random generator. [ 4.618479] random: ln: uninitialized urandom read (6 bytes read) [ 4.730092] random: systemd: uninitialized urandom read (16 bytes read) [ 4.732416] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.740728] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.745049] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.516543] device-mapper: uevent: version 1.0.3 [ 5.518390] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.735341] random: fast init done [ 6.755953] virtio_net virtio0 ens2: renamed from eth0 [ 7.137171] scsi host0: ata_piix [ 7.224149] scsi host1: ata_piix [ 7.225479] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.234597] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.951301] random: crng init done [ 11.952642] random: 7 urandom warning(s) missed due to ratelimiting [ 11.975669] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.819692] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.427652] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.840755] SELinux: Disabled at runtime. [ 14.917414] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.938781] systemd[1]: Detected virtualization kvm. [ 14.947576] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.851839] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.863527] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.870268] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.874958] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.884981] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.899254] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.919583] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ 16.393229] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.112942] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.784956] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.907021] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.103514] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.183127] EDAC sbridge: Ver: 1.1.2 [ 21.190643] Key type dns_resolver registered [ 21.576252] NFS: Registering the id_resolver key type [ 21.578385] Key type id_resolver registered [ 21.580948] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 44.317228] libcfs: loading out-of-tree module taints kernel. [ 44.325519] Key type ._llcrypt registered [ 44.326560] Key type .llcrypt registered [ 44.361043] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 51.498628] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 52.027658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.033518] alg: No test for adler32 (adler32-zlib) [ 52.994533] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 53.261164] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 54.863203] Key type lgssc registered [ 55.400756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.573312] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 62.503629] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 65.976951] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 68.455349] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.875406] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.666951] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 79.690628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.710225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 79.718106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 80.796541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 80.806059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 80.836721] Lustre: lustre-MDT0000: new disk, initializing [ 80.857669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 80.863311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 82.017354] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 87.105660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.132856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 87.161309] Lustre: 6464:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 87.174176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 87.175841] Lustre: Skipped 1 previous similar message [ 87.213910] Lustre: lustre-MDT0001: new disk, initializing [ 87.234179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 87.244898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 87.247242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 88.559291] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.844916] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 94.105851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.133404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 94.225730] Lustre: lustre-OST0000: new disk, initializing [ 94.228710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 94.247767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 96.092486] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.358590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 101.362142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 101.378432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.387859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 101.404517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 101.438311] Lustre: lustre-OST0001: new disk, initializing [ 101.439828] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 101.453426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 103.276497] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.011628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 108.014854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 108.029814] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 109.082574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 113.818443] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 120.700158] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 122.110618] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 123.800899] Lustre: DEBUG MARKER: Client: 2.16.57.62 [ 124.644333] Lustre: DEBUG MARKER: MDS: 2.16.57.62 [ 125.523964] Lustre: DEBUG MARKER: OSS: 2.16.57.62 [ 126.098806] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 19 16:56:32 EDT 2025 [ 131.769483] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 132.227063] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 143.839927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 143.839938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 143.840528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 143.846169] Lustre: Skipped 3 previous similar messages [ 148.959770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.963128] Lustre: Skipped 6 previous similar messages [ 149.816775] LustreError: 12128:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 149.843851] Lustre: server umount lustre-MDT0000 complete [ 150.982896] LustreError: 6454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637017 with bad export cookie 9787979820300560169 [ 150.984081] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 150.986530] LustreError: 6454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 154.079586] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.079879] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 154.084517] Lustre: Skipped 1 previous similar message [ 160.223526] LustreError: 7401:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 160.223677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 160.229304] LustreError: 7401:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 160.231422] Lustre: Skipped 4 previous similar messages [ 161.247706] LustreError: 6469:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 161.251745] LustreError: 6469:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 165.343112] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 165.376102] LustreError: 12331:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 165.378895] LustreError: 12331:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 165.458346] Lustre: server umount lustre-MDT0001 complete [ 166.793053] LustreError: 12533:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 166.795272] LustreError: 12533:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 166.813791] Lustre: server umount lustre-OST0000 complete [ 168.168853] Lustre: server umount lustre-OST0001 complete [ 170.323381] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 173.537357] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 177.055867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.239373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 181.250757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.522036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.909514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.934201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.018048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.029418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.065752] Lustre: lustre-MDT0000: new disk, initializing [ 186.090792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.098177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.464934] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 191.682374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.711170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 191.760653] Lustre: 14900:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 191.869438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 193.221950] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 196.299002] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 197.436657] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 199.692966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.715125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 199.819398] Lustre: lustre-OST0000: new disk, initializing [ 199.820755] Lustre: Skipped 1 previous similar message [ 199.823452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.826115] Lustre: Skipped 2 previous similar messages [ 199.857724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.860945] Lustre: Skipped 1 previous similar message [ 201.531020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.535487] Lustre: Skipped 1 previous similar message [ 201.538033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.573027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.763741] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 204.992206] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 206.453034] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 206.541947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 207.948531] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 208.030830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 211.938092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 211.938216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 211.940143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 211.940148] Lustre: Skipped 2 previous similar messages [ 211.940469] LustreError: Skipped 1 previous similar message [ 211.952192] Lustre: Skipped 1 previous similar message [ 214.764146] LustreError: 17807:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 214.767777] LustreError: 17807:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 214.833571] Lustre: server umount lustre-OST0000 complete [ 217.056863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 217.061642] Lustre: Skipped 1 previous similar message [ 222.176119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 222.177789] Lustre: Skipped 4 previous similar messages [ 222.269523] LustreError: 18009:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 222.271438] LustreError: 18009:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 222.299562] Lustre: server umount lustre-MDT0000 complete [ 223.512251] LustreError: 14891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637089 with bad export cookie 9787979820300561702 [ 223.515918] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 223.517135] LustreError: 14891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 223.649625] Lustre: server umount lustre-MDT0001 complete [ 225.478638] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:58:12 (1755637092) [ 231.798674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.966392] LustreError: 19345:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 231.996667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.327306] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 236.171165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 237.649881] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 239.421308] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 240.662801] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 241.350029] Lustre: Failing over lustre-MDT0000 [ 241.353255] LustreError: 19374:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 241.355117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 241.356663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 241.357191] LustreError: 19374:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 241.360075] Lustre: Skipped 1 previous similar message [ 241.361535] LustreError: 19374:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 241.420034] LustreError: 20676:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 241.421893] LustreError: 20676:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 241.450510] Lustre: server umount lustre-MDT0000 complete [ 246.496510] LustreError: 20050:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 246.500833] LustreError: 20050:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 251.872619] LustreError: 20050:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 251.877821] LustreError: 20050:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 254.645689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 254.686990] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 256.153251] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 259.374438] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 260.065732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 260.068332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 260.073642] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 265.183927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 265.185857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.514254] LustreError: 21650:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 267.515973] LustreError: 21650:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 267.540833] Lustre: server umount lustre-MDT0000 complete [ 268.836067] LustreError: 20059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637134 with bad export cookie 9787979820300563109 [ 268.839796] LustreError: 20059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 268.839869] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 268.981217] Lustre: server umount lustre-MDT0001 complete [ 271.821686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 271.983300] LustreError: 22326:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.009463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.012588] Lustre: Skipped 2 previous similar messages [ 273.317904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 276.240536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 277.752828] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 279.585191] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 280.723464] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 281.415085] Lustre: Failing over lustre-MDT0000 [ 281.418091] LustreError: 22355:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 281.419912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.420409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.420414] Lustre: Skipped 4 previous similar messages [ 281.420778] LustreError: 22355:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 281.424144] Lustre: Skipped 2 previous similar messages [ 281.430054] LustreError: 22355:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 281.512568] Lustre: server umount lustre-MDT0000 complete [ 291.808266] LustreError: 22326:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.815324] LustreError: 22326:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 294.427494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.469218] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.854908] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 298.960940] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 300.000788] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 300.003080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 300.004954] Lustre: Skipped 1 previous similar message [ 300.009911] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 315.359131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.396018] LustreError: 24625:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 315.397859] LustreError: 24625:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 315.421917] Lustre: server umount lustre-MDT0000 complete [ 316.660409] LustreError: 24626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637182 with bad export cookie 9787979820300564033 [ 316.663815] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.664645] LustreError: 24626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.804875] Lustre: server umount lustre-MDT0001 complete [ 319.642406] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:59:46 (1755637186) [ 324.738023] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 327.842271] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 331.315769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.455643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.411233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.474932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.656212] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 344.511980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 344.532811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.621954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 344.634034] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 344.674032] Lustre: lustre-MDT0000: new disk, initializing [ 344.698523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 344.701124] Lustre: Skipped 2 previous similar messages [ 344.706873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 346.120664] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 350.198705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 350.223663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 350.260215] Lustre: 28273:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 350.362560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 351.640331] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 354.698139] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 355.890037] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 358.127449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.147807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 360.067796] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 360.090188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 360.095930] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 360.143743] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 363.315217] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 365.233969] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 370.143821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 370.145950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.148934] Lustre: Skipped 2 previous similar messages [ 370.150728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 370.152234] Lustre: Skipped 8 previous similar messages [ 374.718786] Lustre: server umount lustre-OST0000 complete [ 376.287824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 376.290076] LustreError: Skipped 1 previous similar message [ 382.074385] LustreError: 31096:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 382.076224] LustreError: 31096:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 382.103077] Lustre: server umount lustre-MDT0000 complete [ 383.347357] LustreError: 28265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637249 with bad export cookie 9787979820300564432 [ 383.350370] LustreError: 28265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 383.351771] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.546823] Lustre: server umount lustre-MDT0001 complete [ 387.154861] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 388.275371] Key type lgssc unregistered [ 388.420360] LNet: 31824:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.422494] LNetError: 31824:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.430298] LNet: Removed LNI 192.168.204.146@tcp [ 388.786123] Key type .llcrypt unregistered [ 388.787213] Key type ._llcrypt unregistered [ 397.560906] Key type ._llcrypt registered [ 397.562803] Key type .llcrypt registered [ 397.600742] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 397.893130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 397.916062] alg: No test for adler32 (adler32-zlib) [ 398.764190] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 398.847392] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 400.431143] Key type lgssc registered [ 400.792201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 403.270126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 403.276848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.432276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.800657] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 408.688238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 408.815059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.111471] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 411.943385] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 413.244971] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 415.530716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 415.654218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.548261] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 418.728152] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 419.303712] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 434.755796] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:01:41 (1755637301) [ 435.325796] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 435.895784] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 437.844195] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:01:44 (1755637304) [ 440.294419] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:01:47 (1755637307) [ 440.585027] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 443.359645] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 443.361660] Lustre: Skipped 2 previous similar messages [ 443.937251] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:01:50 (1755637310) [ 447.425747] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 447.428127] Lustre: Skipped 7 previous similar messages [ 451.827831] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:01:58 (1755637318) [ 452.726865] Lustre: *** cfs_fail_loc=131b, val=2*** [ 453.955176] Lustre: *** cfs_fail_loc=131b, val=2*** [ 456.620188] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:02:03 (1755637323) [ 457.275176] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 457.277665] Lustre: Skipped 6 previous similar messages [ 461.332679] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:02:08 (1755637328) [ 465.888893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.890276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.894023] Lustre: Skipped 1 previous similar message [ 465.897395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 471.010956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.013103] Lustre: Skipped 2 previous similar messages [ 471.406098] LustreError: 39752:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 471.483462] Lustre: server umount lustre-OST0000 complete [ 475.615938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 475.618390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.622996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.248660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.250363] Lustre: Skipped 3 previous similar messages [ 486.368132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.370086] Lustre: Skipped 1 previous similar message [ 486.879127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.912720] LustreError: 39953:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 486.914433] LustreError: 39953:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 486.941843] Lustre: server umount lustre-MDT0000 complete [ 488.208324] LustreError: 34807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637354 with bad export cookie 4520754608242230067 [ 488.210418] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.211590] LustreError: 34807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 488.282091] LustreError: 40155:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 488.284145] LustreError: 40155:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 488.367626] Lustre: server umount lustre-MDT0001 complete [ 491.926496] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 492.981480] Key type lgssc unregistered [ 493.109324] LNet: 40681:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.112239] LNetError: 40681:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 493.122324] LNet: Removed LNI 192.168.204.146@tcp [ 493.433147] Key type .llcrypt unregistered [ 493.435133] Key type ._llcrypt unregistered [ 499.333657] Key type ._llcrypt registered [ 499.335547] Key type .llcrypt registered [ 499.377735] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 505.728512] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 506.060704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 506.067167] alg: No test for adler32 (adler32-zlib) [ 506.924406] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 507.006225] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 508.591178] Key type lgssc registered [ 508.947258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 511.805500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 513.921488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 515.935836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 518.055417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 520.298043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 520.319615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 520.325986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.408668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 521.419623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 521.457951] Lustre: lustre-MDT0000: new disk, initializing [ 521.480732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.486980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 522.808304] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 526.905298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 526.927786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 526.961425] Lustre: 43781:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 526.981694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 526.983334] Lustre: Skipped 1 previous similar message [ 527.017797] Lustre: lustre-MDT0001: new disk, initializing [ 527.051630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 527.062930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 527.066117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 528.334318] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 531.423516] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 532.656310] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 534.988747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 535.013975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 535.116717] Lustre: lustre-OST0000: new disk, initializing [ 535.119525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 535.149352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.024607] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 539.639191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 539.642687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 539.672380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 540.129289] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 541.555873] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 541.632310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 542.953937] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 543.033389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 544.737358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.738193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.740596] LustreError: Skipped 1 previous similar message [ 544.741313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.744307] Lustre: Skipped 1 previous similar message [ 544.748044] Lustre: Skipped 1 previous similar message [ 545.709343] LustreError: 46688:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 545.783149] Lustre: server umount lustre-OST0000 complete [ 547.807920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 547.809821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.813307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.856213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.857150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.149510] LustreError: 46888:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 553.152849] LustreError: 46888:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 553.181225] Lustre: server umount lustre-MDT0000 complete [ 554.401304] LustreError: 44555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637420 with bad export cookie 10916589926768301436 [ 554.404083] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.407166] LustreError: 44555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 554.477715] LustreError: 47089:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 554.479829] LustreError: 47089:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 554.562806] Lustre: server umount lustre-MDT0001 complete [ 557.776741] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:03:44 (1755637424) [ 560.026615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.197820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.463753] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 564.284809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 564.407751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 565.747698] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 567.503168] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 568.670099] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 570.864312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 572.807277] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 574.567821] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 592.185530] LustreError: 50008:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 592.520374] Lustre: Modifying parameter general.timeout=1 in log params [ 593.719880] Lustre: Modifying parameter general.timeout=5 in log params [ 593.721676] Lustre: Skipped 3 previous similar messages [ 595.861767] Lustre: Modifying parameter general.timeout=12 in log params [ 595.863816] Lustre: Skipped 6 previous similar messages [ 599.952233] Lustre: Modifying parameter general.timeout=26 in log params [ 599.955157] Lustre: Skipped 13 previous similar messages [ 608.189469] Lustre: Modifying parameter general.timeout=54 in log params [ 608.190807] Lustre: Skipped 27 previous similar messages [ 624.277813] Lustre: Modifying parameter general.timeout=109 in log params [ 624.280541] Lustre: Skipped 54 previous similar messages [ 633.575453] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:05:00 (1755637500) [ 634.545775] Lustre: 56902:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 635.759048] Lustre: 57098:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 635.762516] Lustre: 57098:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 646.207192] Lustre: 58745:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 646.209812] Lustre: 58745:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 3 previous similar messages [ 646.214129] LustreError: 58745:0:(mgs_handler.c:1119:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 647.569952] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:05:14 (1755637514) [ 651.233442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.234388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 651.236652] Lustre: Skipped 2 previous similar messages [ 651.238826] Lustre: Skipped 2 previous similar messages [ 656.352314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 656.354131] Lustre: Skipped 2 previous similar messages [ 657.213459] LustreError: 59442:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 657.215428] LustreError: 59442:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 657.241861] Lustre: server umount lustre-MDT0000 complete [ 658.482095] LustreError: 50058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637524 with bad export cookie 10916589926768302304 [ 658.484876] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.487533] LustreError: 50058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 658.623863] Lustre: server umount lustre-MDT0001 complete [ 669.105616] LustreError: 59846:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 669.107290] LustreError: 59846:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 669.178306] Lustre: server umount lustre-OST0000 complete [ 671.946282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 673.200896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 674.472284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 675.771279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 678.069356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.178027] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 678.188771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 678.193073] Lustre: Skipped 43 previous similar messages [ 678.293036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.295148] Lustre: Skipped 1 previous similar message [ 679.627370] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 682.440302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 682.469135] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 682.479117] Lustre: 61298:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 683.962537] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 685.790163] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 687.009606] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 689.317343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 689.398776] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 691.354536] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 693.103056] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 698.714255] Lustre: 63516:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 704.992616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 704.994365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.994710] LustreError: Skipped 1 previous similar message [ 705.000612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 705.003912] Lustre: Skipped 1 previous similar message [ 706.604078] LustreError: 63761:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 706.606095] LustreError: 63761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 706.667258] Lustre: server umount lustre-OST0000 complete [ 708.063963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 714.011497] Lustre: server umount lustre-MDT0000 complete [ 715.232636] LustreError: 61308:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.237459] LustreError: 61308:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 715.301567] LustreError: 61288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637581 with bad export cookie 10916589926768304593 [ 715.305858] LustreError: 61288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 715.305948] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.444816] Lustre: server umount lustre-MDT0001 complete [ 719.169821] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 720.274170] Key type lgssc unregistered [ 720.400389] LNet: 64696:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.403880] LNetError: 64696:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.414311] LNet: Removed LNI 192.168.204.146@tcp [ 720.721745] Key type .llcrypt unregistered [ 720.723654] Key type ._llcrypt unregistered [ 730.155137] Key type ._llcrypt registered [ 730.156160] Key type .llcrypt registered [ 730.192306] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 730.552622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 730.594643] alg: No test for adler32 (adler32-zlib) [ 731.452862] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 731.547385] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 733.127148] Key type lgssc registered [ 733.501176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 736.169506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 736.176982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.345895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.682857] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 741.606630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 741.818122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 743.194761] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 745.009143] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 746.214902] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 748.491235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 748.669569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.606323] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 752.470410] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 770.811257] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 773.600622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.602180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.604274] Lustre: Skipped 1 previous similar message [ 778.722083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 778.724038] Lustre: Skipped 4 previous similar messages [ 779.329883] LustreError: 68770:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 779.356065] Lustre: server umount lustre-MDT0000 complete [ 780.600412] LustreError: 66041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637646 with bad export cookie 15794004549618565092 [ 780.604477] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 780.606257] LustreError: 66041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 780.681781] LustreError: 68972:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 780.684226] LustreError: 68972:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 780.771916] Lustre: server umount lustre-MDT0001 complete [ 801.247810] LustreError: 69173:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 801.249841] LustreError: 69173:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 801.316735] Lustre: server umount lustre-OST0000 complete [ 804.101334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 805.377042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 806.659235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 807.968902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 810.260122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.352687] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 810.363736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 810.448910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.732673] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 814.497671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 814.527438] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 814.537762] Lustre: 70621:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 814.708572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 816.018675] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 817.801716] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 819.057192] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 821.367693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 821.438254] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 823.582674] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 825.367383] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 831.317590] Lustre: 72845:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 831.941491] Lustre: Modifying parameter general.jobid_var=disable in log params [ 833.508298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 833.508371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.510982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 833.514533] Lustre: Skipped 2 previous similar messages [ 835.246906] LustreError: 73094:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 835.249247] LustreError: 73094:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 835.315244] Lustre: server umount lustre-OST0000 complete [ 838.624807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.629827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.632178] Lustre: Skipped 2 previous similar messages [ 838.682575] Lustre: server umount lustre-MDT0000 complete [ 839.926372] LustreError: 72847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637705 with bad export cookie 15794004549618565890 [ 839.929889] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 839.930197] LustreError: 72847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 839.994868] LustreError: 73498:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 839.996823] LustreError: 73498:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 840.074993] Lustre: server umount lustre-MDT0001 complete [ 843.983783] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 845.061552] Key type lgssc unregistered [ 845.200455] LNet: 74024:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.205287] LNetError: 74024:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.212328] LNet: Removed LNI 192.168.204.146@tcp [ 845.519556] Key type .llcrypt unregistered [ 845.521128] Key type ._llcrypt unregistered [ 854.916414] Key type ._llcrypt registered [ 854.918034] Key type .llcrypt registered [ 854.956309] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 855.351496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.360213] alg: No test for adler32 (adler32-zlib) [ 856.216862] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 856.305774] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 857.887143] Key type lgssc registered [ 858.242852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 860.769945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.778807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.949929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.223355] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 865.947683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 866.143603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 867.531690] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 869.362279] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 870.607768] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 872.887215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 873.067435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.874639] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 876.614389] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 889.399554] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 890.014332] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 890.016710] Lustre: Skipped 1 previous similar message [ 891.288232] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 891.289825] Lustre: Skipped 3 previous similar messages [ 893.422981] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 893.424756] Lustre: Skipped 6 previous similar messages [ 897.655389] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 897.657945] Lustre: Skipped 12 previous similar messages [ 905.903337] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 905.906207] Lustre: Skipped 27 previous similar messages [ 922.254648] Lustre: 83123:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 922.578596] Lustre: Disabling parameter general.jobid_var= in log params [ 922.580854] Lustre: Skipped 45 previous similar messages [ 923.182952] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:09:49 (1755637789) [ 923.753915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 924.320388] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:09:51 (1755637791) [ 927.715432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.715601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.716819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 927.718537] LustreError: Skipped 1 previous similar message [ 927.723860] Lustre: Skipped 1 previous similar message [ 931.371140] LustreError: 83418:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 931.433695] Lustre: server umount lustre-OST0000 complete [ 932.832569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.832583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 932.833155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.833160] Lustre: Skipped 1 previous similar message [ 932.836986] Lustre: Skipped 1 previous similar message [ 937.952429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 937.954931] Lustre: Skipped 2 previous similar messages [ 938.813515] LustreError: 83620:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 938.815735] LustreError: 83620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 938.845275] Lustre: server umount lustre-MDT0000 complete [ 940.088264] LustreError: 75368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637806 with bad export cookie 3853932521512802612 [ 940.091556] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.092746] LustreError: 75368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 940.161799] LustreError: 83822:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 940.164030] LustreError: 83822:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 940.252096] Lustre: server umount lustre-MDT0001 complete [ 943.834104] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 944.899439] Key type lgssc unregistered [ 945.043403] LNet: 84348:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.045547] LNetError: 84348:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.053498] LNet: Removed LNI 192.168.204.146@tcp [ 945.391772] Key type .llcrypt unregistered [ 945.392949] Key type ._llcrypt unregistered [ 947.027296] Key type ._llcrypt registered [ 947.028191] Key type .llcrypt registered [ 947.067436] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 948.444514] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules [ 948.827476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.883803] alg: No test for adler32 (adler32-zlib) [ 949.743173] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 949.824589] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 951.407157] Key type lgssc registered [ 951.870055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.292587] LustreError: 85452:0:(lu_object.c:1390:lu_context_key_register()) cfs_race id 60d sleeping [ 959.693938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 962.932845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.422671] LustreError: 85452:0:(lu_object.c:1390:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 973.430975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.606609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.867547] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 978.569791] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:10:45 (1755637845) [ 980.162817] LustreError: 86485:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 980.166533] LustreError: 86485:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 980.169524] LustreError: 86485:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 980.233467] LustreError: 87252:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 980.331963] Lustre: server umount lustre-MDT0000 complete [ 984.948590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 986.101729] Key type lgssc unregistered [ 986.247337] LNet: 87932:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.249569] LNetError: 87932:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.256373] LNet: Removed LNI 192.168.204.146@tcp [ 986.569949] Key type .llcrypt unregistered [ 986.571122] Key type ._llcrypt unregistered [ 993.488195] Key type ._llcrypt registered [ 993.489287] Key type .llcrypt registered [ 993.524725] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 993.845500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 993.891087] alg: No test for adler32 (adler32-zlib) [ 994.749140] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 994.831542] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 996.415135] Key type lgssc registered [ 996.756659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.231340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.237560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.400057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.618388] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1004.393869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1004.613177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.872830] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1007.530025] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1008.746799] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1011.009847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1011.220972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.188901] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1014.977257] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1051.881537] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:11:58 (1755637918) [ 1052.422772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1052.974369] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:11:59 (1755637919) [ 1055.711812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1055.714510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.720316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.224522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.226104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.027068] LustreError: 92015:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1060.052334] Lustre: server umount lustre-MDT0000 complete [ 1061.289367] LustreError: 89083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637927 with bad export cookie 3813857088028012444 [ 1061.292875] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.294202] LustreError: 89083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1061.343832] LustreError: 89097:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.344087] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.349911] LustreError: 89097:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1061.353675] Lustre: Skipped 1 previous similar message [ 1061.354499] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1061.359792] Lustre: Skipped 2 previous similar messages [ 1061.360079] LustreError: 92218:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1061.362438] LustreError: 92218:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1061.439081] Lustre: server umount lustre-MDT0001 complete [ 1068.779210] LustreError: 92418:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1068.781292] LustreError: 92418:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1068.936718] Lustre: server umount lustre-OST0000 complete [ 1073.451031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.625625] LustreError: 93047:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.683188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.968340] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1077.730854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1077.950094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.202092] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1080.932406] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1082.169219] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1084.181791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1086.346376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1086.375799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1086.443432] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1086.445840] LustreError: 93036:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1086.454480] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1086.457278] LustreError: 94895:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -98 [ 1086.459372] LustreError: 94895:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 1086.462996] LustreError: 94895:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1086.517511] Lustre: server umount lustre-OST0000 complete [ 1086.518924] LustreError: 94895:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1089.278051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1089.347395] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1089.352370] LustreError: 93035:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1089.357588] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1089.361060] LustreError: 95358:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -98 [ 1089.363095] LustreError: 95358:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 1089.365274] LustreError: 95358:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1089.415920] Lustre: server umount lustre-OST0000 complete [ 1089.418076] LustreError: 95358:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1090.346057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1092.579097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1092.642823] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1092.644469] Lustre: Found index 0 for lustre-OST0000, updating log [ 1092.647670] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1092.677333] Lustre: lustre-OST0000: new disk, initializing [ 1092.679272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1092.770420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.664812] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1095.876708] Lustre: Failing over lustre-OST0000 [ 1095.894069] LustreError: 96428:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1095.895753] LustreError: 96428:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1095.955236] Lustre: server umount lustre-OST0000 complete [ 1097.136845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.137412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.139888] Lustre: Skipped 1 previous similar message [ 1102.305305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.307300] Lustre: Skipped 2 previous similar messages [ 1103.225604] LustreError: 96629:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1103.227268] LustreError: 96629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1103.252482] Lustre: server umount lustre-MDT0000 complete [ 1104.435458] LustreError: 93028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637970 with bad export cookie 3813857088028015174 [ 1104.438419] LustreError: 93028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1104.438523] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1104.566880] Lustre: server umount lustre-MDT0001 complete [ 1107.468784] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:12:54 (1755637974) [ 1112.753654] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1116.324309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.454305] LustreError: 98195:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.482889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.743435] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1120.495246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1122.016688] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1122.985040] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1125.145983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1127.217594] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1129.987350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1130.015854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1130.039189] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1130.072029] Lustre: lustre-OST0001: new disk, initializing [ 1130.073644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1131.967950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1131.973209] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1132.012293] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1132.020574] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1135.185175] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1140.888585] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1147.362284] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.363414] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1147.365173] Lustre: Skipped 1 previous similar message [ 1147.368712] Lustre: Skipped 2 previous similar messages [ 1153.274934] LustreError: 101411:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1153.277818] LustreError: 101411:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1153.301236] Lustre: server umount lustre-MDT0001 complete [ 1153.523182] LustreError: 99831:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.258670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1156.474159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1156.507427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1156.509230] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1156.522479] Lustre: 98184:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1156.751378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.753499] Lustre: Skipped 3 previous similar messages [ 1157.861350] LustreError: 102127:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1157.863531] LustreError: 102127:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1157.865803] LustreError: 98186:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1157.894587] LustreError: 102134:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -17 [ 1157.897876] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1157.897876] [ 1158.077929] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1161.290261] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 17:13:48 (1755638028) [ 1162.216822] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1166.891775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1172.716374] Lustre: 104003:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1180.998374] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1182.261633] Lustre: 105134:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1233.375821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1233.379174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.382918] Lustre: Skipped 1 previous similar message [ 1233.385253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.386947] Lustre: Skipped 4 previous similar messages [ 1239.496982] LustreError: 105569:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1239.498728] LustreError: 105569:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1239.523835] Lustre: server umount lustre-MDT0000 complete [ 1240.842638] LustreError: 105469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638106 with bad export cookie 3813857088028016042 [ 1240.844330] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.845824] LustreError: 105469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1240.995359] Lustre: server umount lustre-MDT0001 complete [ 1252.365752] Lustre: server umount lustre-OST0000 complete [ 1263.932743] Lustre: server umount lustre-OST0001 complete [ 1265.914674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1268.803771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1269.124396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1271.868665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1274.784983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1275.122644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1281.528314] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1284.937490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.945778] Lustre: lustre-MDT0000: reset Object Index mappings [ 1285.105558] LustreError: 109040:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.131628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.400823] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1289.223869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1289.230843] Lustre: lustre-MDT0001: reset Object Index mappings [ 1290.910634] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1291.901373] Lustre: 110143:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1294.131591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1295.394464] LustreError: 110499:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.400266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 1296.326083] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1299.121981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1300.332342] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1300.332685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1300.333636] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1301.153913] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1304.277121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1305.440706] Lustre: 111987:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1325.398927] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:16:32 (1755638192) [ 1356.255995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1356.258463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.262584] Lustre: Skipped 3 previous similar messages [ 1356.264774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.267146] Lustre: Skipped 7 previous similar messages [ 1360.773263] LustreError: 112335:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1360.775008] LustreError: 112335:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1360.804323] Lustre: server umount lustre-MDT0000 complete [ 1361.890143] LustreError: 109039:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.894015] LustreError: 109039:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1362.022469] LustreError: 111989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638228 with bad export cookie 3813857088028062767 [ 1362.026755] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.027348] LustreError: 111989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1368.343730] Lustre: server umount lustre-MDT0001 complete [ 1375.754835] Lustre: server umount lustre-OST0000 complete [ 1383.150712] Lustre: server umount lustre-OST0001 complete [ 1385.194503] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1388.214632] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1391.861788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1393.979536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.875159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1397.957428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.178130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.201280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.292384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1400.303208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1400.338455] Lustre: lustre-MDT0000: new disk, initializing [ 1400.368680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1401.651844] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1404.658606] Lustre: server umount lustre-MDT0000 complete [ 1405.584913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1407.778954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1407.800484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.889145] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1407.893899] Lustre: 116197:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1407.897673] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1409.283331] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1410.919689] Lustre: server umount lustre-MDT0000 complete [ 1413.768363] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:18:00 (1755638280) [ 1414.305942] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1414.870111] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:18:01 (1755638281) [ 1420.102375] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1423.423028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.564273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.566302] Lustre: Skipped 5 previous similar messages [ 1424.884266] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1427.817681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1427.845177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1427.876967] Lustre: 118203:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1427.899224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1427.901675] Lustre: Skipped 1 previous similar message [ 1427.939223] Lustre: lustre-MDT0001: new disk, initializing [ 1427.986879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1427.991380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1429.267422] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1431.514409] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1433.679175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1433.700812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1433.790348] Lustre: lustre-OST0000: new disk, initializing [ 1433.791885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1435.671440] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1435.832457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1435.835832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1435.872328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1439.778839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1439.804046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1441.107955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1441.111884] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1441.152678] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1441.745679] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1446.078738] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1447.235499] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1569.582067] Lustre: 126938:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1569.585636] Lustre: 126938:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1569.588069] Lustre: 126938:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1569.590849] Lustre: 126938:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1569.593767] Lustre: 126938:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1569.596769] Lustre: 126938:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1637.975043] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:21:44 (1755638504) [ 1651.170350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.176512] Lustre: Skipped 6 previous similar messages [ 1651.178413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.181362] Lustre: Skipped 7 previous similar messages [ 1655.552519] LustreError: 127145:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1655.554408] LustreError: 127145:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1655.612192] Lustre: server umount lustre-MDT0000 complete [ 1656.293043] LustreError: 118207:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.298428] LustreError: 118207:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 1656.880568] LustreError: 118194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638522 with bad export cookie 3813857088028116030 [ 1656.884602] LustreError: 118194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1656.885681] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.170722] Lustre: server umount lustre-MDT0001 complete [ 1670.723212] Lustre: server umount lustre-OST0000 complete [ 1678.055457] Lustre: server umount lustre-OST0001 complete [ 1680.025262] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1682.964362] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1686.280192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1688.440915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1690.447784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1692.587275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1694.803275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1694.826057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.909958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1694.920183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1694.921817] Lustre: Skipped 1 previous similar message [ 1694.957566] Lustre: lustre-MDT0000: new disk, initializing [ 1694.958765] Lustre: Skipped 1 previous similar message [ 1694.984599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.988098] Lustre: Skipped 3 previous similar messages [ 1694.996143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1696.250312] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1700.382556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1700.405124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1700.440464] Lustre: 129918:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1700.541273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1701.794227] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1704.872824] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1706.157463] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1708.354987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1708.383026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1710.202909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1710.242571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1710.370638] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1713.346340] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1717.239769] Lustre: lustre-MDD0000: changelog on [ 1718.307556] Lustre: lustre-MDD0001: changelog on [ 2149.316921] Lustre: lustre-MDD0001: changelog off [ 2149.941747] Lustre: 129923:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2150.310440] Lustre: lustre-MDD0000: changelog off [ 2153.958616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.959903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.965300] Lustre: Skipped 4 previous similar messages [ 2153.970952] Lustre: Skipped 3 previous similar messages [ 2158.187702] LustreError: 139133:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2158.190308] LustreError: 139133:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2158.260196] Lustre: server umount lustre-OST0000 complete [ 2161.632646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2165.949872] Lustre: server umount lustre-MDT0000 complete [ 2167.633226] LustreError: 129910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639033 with bad export cookie 3813857088028881410 [ 2167.639359] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.639558] LustreError: 129910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2167.821579] Lustre: server umount lustre-MDT0001 complete [ 2171.571972] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2172.585777] Key type lgssc unregistered [ 2172.711382] LNet: 140061:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.715472] LNetError: 140061:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.726304] LNet: Removed LNI 192.168.204.146@tcp [ 2173.020119] Key type .llcrypt unregistered [ 2173.021073] Key type ._llcrypt unregistered [ 2179.774296] Key type ._llcrypt registered [ 2179.775415] Key type .llcrypt registered [ 2179.809261] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2186.125855] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2186.397759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2186.404546] alg: No test for adler32 (adler32-zlib) [ 2187.241681] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2187.315057] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2188.887156] Key type lgssc registered [ 2189.295322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2193.162890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2195.682959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2197.834384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2200.275383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2203.089504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2203.118137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2203.126608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.215708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2204.228422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2204.264514] Lustre: lustre-MDT0000: new disk, initializing [ 2204.289117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.295270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2205.974621] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2211.044599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2211.085929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2211.135245] Lustre: 143199:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2211.174698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2211.178310] Lustre: Skipped 1 previous similar message [ 2211.222530] Lustre: lustre-MDT0001: new disk, initializing [ 2211.258609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2211.272047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2211.276636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2212.690507] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2216.278935] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2217.598737] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2220.196342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2220.219731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2220.317235] Lustre: lustre-OST0000: new disk, initializing [ 2220.319773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2220.352878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.146972] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2225.816983] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2227.190339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2227.192895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2227.223061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2233.351676] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module obdecho/obdecho [ 2234.068611] LustreError: 145796:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2234.073410] LustreError: 145796:0:(obd_config.c:779:class_setup()) setup ec failed (-22) [ 2234.654986] LustreError: 145892:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2242.016800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2242.021137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.028819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.532488] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.540747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.544421] Lustre: Skipped 1 previous similar message [ 2244.356335] LustreError: 146231:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2244.390198] Lustre: server umount lustre-MDT0000 complete [ 2245.812501] LustreError: 143192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639111 with bad export cookie 2849379363951350115 [ 2245.818754] LustreError: 143192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2245.819601] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2245.908188] LustreError: 146433:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2245.913329] LustreError: 146433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2246.028573] Lustre: server umount lustre-MDT0001 complete [ 2257.718388] LustreError: 146634:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2257.721386] LustreError: 146634:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2257.812420] Lustre: server umount lustre-OST0000 complete [ 2266.985215] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2271.350716] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2274.838054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2277.042051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2279.615304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2282.144926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2285.015815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2285.055852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.170353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2285.179612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2285.222834] Lustre: lustre-MDT0000: new disk, initializing [ 2285.246476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.251974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2286.589496] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2290.890536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2290.912680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2290.941405] Lustre: 149564:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2290.976560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2290.977613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2290.978475] Lustre: Skipped 1 previous similar message [ 2291.017136] Lustre: lustre-MDT0001: new disk, initializing [ 2291.048762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2291.060612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2291.063505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2292.496343] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2295.533258] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2296.706459] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2298.781620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2298.803432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2300.650234] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2303.543735] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2304.764328] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2306.553261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2306.556555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2306.591060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2306.885065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2308.203319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2308.282822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2311.652797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2311.652921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.655718] LustreError: Skipped 1 previous similar message [ 2311.660044] Lustre: Skipped 1 previous similar message [ 2311.661031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2311.664901] Lustre: Skipped 1 previous similar message [ 2315.053086] LustreError: 152487:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2315.054799] LustreError: 152487:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2315.116334] Lustre: server umount lustre-OST0000 complete [ 2316.768666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.768887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2316.769368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.772802] Lustre: Skipped 2 previous similar messages [ 2322.361893] LustreError: 152689:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2322.363516] LustreError: 152689:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2322.391161] Lustre: server umount lustre-MDT0000 complete [ 2324.138610] LustreError: 149554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639190 with bad export cookie 2849379363951351060 [ 2324.144937] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2324.148564] LustreError: 149554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2324.325886] Lustre: server umount lustre-MDT0001 complete [ 2325.988960] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:33:12 (1755639192) [ 2326.653593] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2327.456246] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:33:14 (1755639194) [ 2330.803930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2330.964595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.966451] Lustre: Skipped 1 previous similar message [ 2332.456412] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2335.888698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2337.268637] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2339.728455] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2341.416461] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2344.171438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2346.706397] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2348.898052] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2361.824178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2361.826059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.829180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.830578] Lustre: Skipped 3 previous similar messages [ 2367.233492] LustreError: 155811:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2367.235225] LustreError: 155811:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2367.264282] Lustre: server umount lustre-MDT0000 complete [ 2368.486285] LustreError: 153540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639234 with bad export cookie 2849379363951351942 [ 2368.489409] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2368.489721] LustreError: 153540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2369.525194] LustreError: 153556:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.535421] LustreError: 153556:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2371.040327] LustreError: 153556:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.040598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.051609] Lustre: Skipped 2 previous similar messages [ 2373.088397] LustreError: 153556:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.719130] Lustre: server umount lustre-MDT0001 complete [ 2377.892579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2383.285651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2383.405107] LustreError: 157074:0:(llog.c:1602:llog_backup()) MGC192.168.204.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2383.409479] Lustre: 157074:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2388.978933] LustreError: 157082:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.100319] LustreError: 157082:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.109778] LustreError: 157082:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2403.809404] LustreError: 157082:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.818459] LustreError: 157082:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 2420.723721] LustreError: 157096:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.733657] LustreError: 157096:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [ 2444.767338] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2444.771069] Lustre: 157074:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2453.984130] LustreError: 157082:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.993168] LustreError: 157082:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 21 previous similar messages [ 2473.951296] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2473.954461] Lustre: 153552:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2485.229902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2485.234309] Lustre: Skipped 2 previous similar messages [ 2486.861671] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2488.292377] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 2489.770324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.431210] LustreError: 153552:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2513.890321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 2513.892379] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x0 to 0x278b076f52a631ca [ 2513.895119] Lustre: MGC192.168.204.146@tcp: Connection restored to (at 0@lo) [ 2515.216314] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2516.859659] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2517.368260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2518.430880] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2519.013678] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2522.081480] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2527.645326] LustreError: 158550:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2527.647840] LustreError: 158550:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2527.677170] Lustre: server umount lustre-MDT0000 complete [ 2529.067255] LustreError: 153540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639395 with bad export cookie 2849379363951358410 [ 2529.070228] LustreError: 153540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2529.071115] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.195156] Lustre: server umount lustre-MDT0001 complete [ 2551.037541] Lustre: server umount lustre-OST0000 complete [ 2559.451882] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2563.504393] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2567.843134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2571.017098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2573.373218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2576.605119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2579.948295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2579.971529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.060168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2580.073954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2580.076009] Lustre: Skipped 1 previous similar message [ 2580.109609] Lustre: lustre-MDT0000: new disk, initializing [ 2580.110774] Lustre: Skipped 1 previous similar message [ 2580.133188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.134827] Lustre: Skipped 1 previous similar message [ 2580.140268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2581.755035] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2587.022083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2587.058620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2587.108700] Lustre: 161885:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2587.233295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2588.879345] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2592.249619] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2593.594940] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2595.955701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2595.988706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2596.115305] Lustre: lustre-OST0000: new disk, initializing [ 2596.118860] Lustre: Skipped 1 previous similar message [ 2596.122930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2596.127551] Lustre: Skipped 2 previous similar messages [ 2598.334562] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2602.132248] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2603.530338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2603.582094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2603.840449] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2603.946458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2605.702122] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2605.825710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2608.612687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2608.612782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.615947] LustreError: Skipped 1 previous similar message [ 2608.621681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2608.624897] Lustre: Skipped 8 previous similar messages [ 2612.912195] LustreError: 164795:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2612.916162] LustreError: 164795:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2613.024005] Lustre: server umount lustre-OST0000 complete [ 2621.020848] Lustre: server umount lustre-MDT0000 complete [ 2622.494719] LustreError: 161876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639488 with bad export cookie 2849379363951375224 [ 2622.497859] LustreError: 161876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2622.498999] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.658216] Lustre: server umount lustre-MDT0001 complete [ 2626.654632] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:38:13 (1755639493) [ 2629.678643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.894577] LustreError: 165872:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.900441] LustreError: 165872:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 45 previous similar messages [ 2631.437114] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2635.126343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2636.986359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2639.389774] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2641.242077] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2644.375212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2644.546524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.548718] Lustre: Skipped 4 previous similar messages [ 2647.153043] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2649.782251] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2659.870273] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:38:46 (1755639526) [ 2661.344604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2661.350029] LustreError: Skipped 1 previous similar message [ 2661.353930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.361388] Lustre: Skipped 3 previous similar messages [ 2661.365514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.370195] Lustre: Skipped 4 previous similar messages [ 2667.248212] Lustre: server umount lustre-OST0000 complete [ 2683.359128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2683.408117] LustreError: 168497:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2683.410623] LustreError: 168497:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 2683.451942] Lustre: server umount lustre-MDT0000 complete [ 2685.333540] LustreError: 165897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639551 with bad export cookie 2849379363951376092 [ 2685.339857] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.567417] Lustre: server umount lustre-MDT0001 complete [ 2689.416777] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2690.845905] Key type lgssc unregistered [ 2691.035644] LNet: 169227:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2691.039347] LNetError: 169227:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2691.050835] LNet: Removed LNI 192.168.204.146@tcp [ 2691.537855] Key type .llcrypt unregistered [ 2691.539207] Key type ._llcrypt unregistered [ 2699.538137] Key type ._llcrypt registered [ 2699.539051] Key type .llcrypt registered [ 2699.576833] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2699.888225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2699.918764] alg: No test for adler32 (adler32-zlib) [ 2700.789867] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2700.880776] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2702.463183] Key type lgssc registered [ 2703.037876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2705.980402] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2705.993085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2723.551333] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2729.695271] LustreError: 170420:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2735.839521] LustreError: 170420:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2735.844116] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2735.846458] LustreError: MGC192.168.204.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2735.850208] LustreError: 170420:0:(tgt_mount.c:1669:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2735.853391] LustreError: 170420:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -22 [ 2735.856203] LustreError: 170420:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 2735.858877] LustreError: 170420:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2735.915680] Lustre: server umount lustre-OST0000 complete [ 2735.917837] LustreError: 170420:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2739.822209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.037218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.959977] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2746.739454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2746.866240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.209192] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2750.693336] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2752.247153] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2755.284998] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2755.383394] LustreError: 172498:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2755.388089] Lustre: 172498:0:(mgc_request_server.c:610:mgc_llog_local_copy()) MGC192.168.204.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2755.456068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.584789] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2759.704927] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2761.697584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.698902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.700586] Lustre: Skipped 2 previous similar messages [ 2761.702322] Lustre: Skipped 1 previous similar message [ 2766.821173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.826958] Lustre: Skipped 2 previous similar messages [ 2771.939874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.943530] Lustre: Skipped 1 previous similar message [ 2775.007340] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2775.047708] LustreError: 173143:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2775.051481] LustreError: 173143:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2775.086626] Lustre: server umount lustre-MDT0000 complete [ 2776.782881] LustreError: 170900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639642 with bad export cookie 5089644341236785396 [ 2776.787920] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.789761] LustreError: 170900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2776.894923] LustreError: 173347:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2776.898046] LustreError: 173347:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2777.019475] Lustre: server umount lustre-MDT0001 complete [ 2789.239884] LustreError: 173547:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2789.245281] LustreError: 173547:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2789.368823] Lustre: server umount lustre-OST0000 complete [ 2792.962139] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2797.172033] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2801.539364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2804.969275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2808.125871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2811.480134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.900346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.941092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.081170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2815.103460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2815.156338] Lustre: lustre-MDT0000: new disk, initializing [ 2815.195291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.208347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2817.284857] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2822.873558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2822.912669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2822.973519] Lustre: 175872:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2823.014756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2823.017325] Lustre: Skipped 1 previous similar message [ 2823.063079] Lustre: lustre-MDT0001: new disk, initializing [ 2823.122261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2823.145519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2823.152350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2824.950267] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2828.998774] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2830.432039] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2833.402850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2833.437471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2833.558873] Lustre: lustre-OST0000: new disk, initializing [ 2833.560514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2833.596122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.128429] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2839.546330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2839.550882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2839.599505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2840.101134] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2841.931989] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2842.019393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2843.591981] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2843.714345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2844.642558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.645756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.646812] LustreError: Skipped 1 previous similar message [ 2844.654750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.659568] Lustre: Skipped 2 previous similar messages [ 2849.764858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.768282] Lustre: Skipped 1 previous similar message [ 2850.734163] LustreError: 178784:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2850.738082] LustreError: 178784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2850.817558] Lustre: server umount lustre-OST0000 complete [ 2853.856645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.861103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.867928] Lustre: Skipped 1 previous similar message [ 2858.736617] Lustre: server umount lustre-MDT0000 complete [ 2860.000814] LustreError: 175883:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.010589] LustreError: 175883:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2860.502183] LustreError: 175863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639726 with bad export cookie 5089644341236786096 [ 2860.509309] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.511640] LustreError: 175863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2860.602191] LustreError: 179187:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2860.605444] LustreError: 179187:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2860.696882] Lustre: server umount lustre-MDT0001 complete [ 2864.614333] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:42:11 (1755639731) [ 2872.092739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2873.618218] Key type lgssc unregistered [ 2873.795578] LNet: 180365:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2873.800435] LNetError: 180365:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2873.811567] LNet: Removed LNI 192.168.204.146@tcp [ 2874.315640] Key type .llcrypt unregistered [ 2874.317598] Key type ._llcrypt unregistered [ 2883.967288] Key type ._llcrypt registered [ 2883.969654] Key type .llcrypt registered [ 2884.035665] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2884.475861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2884.524774] alg: No test for adler32 (adler32-zlib) [ 2885.450750] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2885.544211] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2887.151305] Key type lgssc registered [ 2887.791436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2891.722919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2891.736874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.995106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.859644] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2898.988078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2899.172908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2901.052374] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2903.774710] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2905.374718] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2909.109543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2909.219378] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2909.220569] LustreError: 183161:0:(llog.c:1642:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2909.222812] Lustre: 183161:0:(mgc_request_server.c:607:mgc_llog_local_copy()) MGC192.168.204.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2909.264780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2911.740462] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2914.197454] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2915.811184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.813809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2915.818030] Lustre: Skipped 2 previous similar messages [ 2915.823320] Lustre: Skipped 1 previous similar message [ 2920.931305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2920.935614] Lustre: Skipped 3 previous similar messages [ 2921.357202] LustreError: 183804:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2921.394194] Lustre: server umount lustre-MDT0000 complete [ 2923.191562] LustreError: 183190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639789 with bad export cookie 15276032256489880301 [ 2923.197200] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.198057] LustreError: 183190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2923.292307] LustreError: 184007:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2923.296124] LustreError: 184007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2923.391955] Lustre: server umount lustre-MDT0001 complete [ 2935.478860] LustreError: 184207:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2935.482595] LustreError: 184207:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2935.583409] Lustre: server umount lustre-OST0000 complete [ 2939.983866] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2944.884767] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2949.462876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2953.059776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2956.321623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2959.719766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.175463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.207141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.339819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2963.357350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2963.415270] Lustre: lustre-MDT0000: new disk, initializing [ 2963.448707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.455924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2965.078359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2970.723311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2970.760660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2970.808226] Lustre: 186531:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2970.837106] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2970.839066] Lustre: Skipped 1 previous similar message [ 2970.841116] LustreError: 187290:0:(llog.c:1642:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2970.845136] LustreError: 187290:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 2970.848054] Lustre: 187290:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2970.857915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2970.860313] Lustre: Skipped 1 previous similar message [ 2970.900929] Lustre: lustre-MDT0001: new disk, initializing [ 2970.944368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2970.960648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2970.965426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2972.611486] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2976.556252] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2978.203562] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2981.037239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2981.061125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2981.160816] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2981.162071] Lustre: Skipped 1 previous similar message [ 2981.163285] LustreError: 188239:0:(llog.c:1642:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2981.166571] LustreError: 188239:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 2981.168467] Lustre: 188239:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2981.171145] Lustre: 188239:0:(mgc_request_server.c:630:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2981.180667] Lustre: lustre-OST0000: new disk, initializing [ 2981.182571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2981.226479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.958144] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2985.487228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2985.493779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2985.559585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2988.091815] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2990.082807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2990.228326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2992.251915] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2992.398613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2995.684915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2995.687988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.688244] LustreError: Skipped 1 previous similar message [ 2995.694063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.698438] Lustre: Skipped 1 previous similar message [ 2999.409114] LustreError: 189439:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2999.412929] LustreError: 189439:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2999.504425] Lustre: server umount lustre-OST0000 complete [ 3001.824918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3001.829229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.836061] Lustre: Skipped 1 previous similar message [ 3001.839277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.922627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.923351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.568133] LustreError: 189641:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3007.572116] LustreError: 189641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3007.612874] Lustre: server umount lustre-MDT0000 complete [ 3009.376625] LustreError: 186523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639875 with bad export cookie 15276032256489880987 [ 3009.381542] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3009.383147] LustreError: 186523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3009.553433] Lustre: server umount lustre-MDT0001 complete [ 3013.505172] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:44:39 (1755639879) [ 3021.008137] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3025.711545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.947195] LustreError: 191202:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.958370] LustreError: 191202:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3025.974719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.849594] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3031.009750] LustreError: 191202:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.290408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3034.286319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3035.796279] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3039.148777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3042.065367] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3046.247904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3046.291681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3046.377890] Lustre: lustre-OST0001: new disk, initializing [ 3046.380604] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3049.025379] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3051.562177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3051.569242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3051.633369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3055.425571] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3060.784788] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3066.632110] Lustre: DEBUG MARKER: ADD OST3 [ 3069.146730] loop: module loaded [ 3069.180341] loop0: detected capacity change from 0 to 204800000 [ 3069.267688] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3069.271929] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3069.276405] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3069.327971] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3072.136094] loop0: detected capacity change from 0 to 204800000 [ 3074.675824] Lustre: DEBUG MARKER: STOP OST3 [ 3074.843854] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3074.871546] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3074.962594] Lustre: lustre-OST0003: new disk, initializing [ 3074.965139] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3075.018897] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3075.023309] Lustre: Skipped 3 previous similar messages [ 3075.563146] Lustre: Failing over lustre-OST0003 [ 3075.585327] LustreError: 195360:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3075.588435] LustreError: 195360:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3075.612413] Lustre: server umount lustre-OST0003 complete [ 3076.435488] LustreError: 192645:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.436270] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3078.772404] LustreError: 193951:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.786436] LustreError: 193951:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3083.241364] Lustre: DEBUG MARKER: START OST3 again [ 3086.606085] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3087.927767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3087.935929] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3087.947028] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3088.986532] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3104.071125] Lustre: Failing over lustre-OST0003 [ 3104.110210] LustreError: 197609:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 3104.113880] LustreError: 197609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3104.139216] Lustre: server umount lustre-OST0003 complete [ 3107.318391] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.320759] LustreError: 192644:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.324887] Lustre: Skipped 1 previous similar message [ 3107.335605] LustreError: 192644:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3117.542455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.551151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.554505] Lustre: Skipped 1 previous similar message [ 3117.557126] LustreError: 192645:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3117.565510] LustreError: 192645:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3123.777879] Lustre: server umount lustre-MDT0000 complete [ 3125.458729] LustreError: 191181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639991 with bad export cookie 15276032256489881869 [ 3125.460490] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3125.465138] LustreError: 191181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3131.830105] Lustre: server umount lustre-MDT0001 complete [ 3133.522651] Lustre: server umount lustre-OST0000 complete [ 3135.210457] Lustre: server umount lustre-OST0001 complete [ 3137.654469] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3140.646145] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3145.287253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3148.474248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3151.456164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3154.386166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3157.778069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3157.818651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.964504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3157.987864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3158.042196] Lustre: lustre-MDT0000: new disk, initializing [ 3158.087902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.090933] Lustre: Skipped 1 previous similar message [ 3158.100556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3159.932358] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3165.517784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3165.548851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3165.590661] Lustre: 200655:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3165.766834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3167.575539] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3171.513703] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3173.218370] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3176.650569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3176.695875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3176.860975] Lustre: lustre-OST0000: new disk, initializing [ 3176.862676] Lustre: Skipped 1 previous similar message [ 3176.864809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3176.866907] Lustre: Skipped 2 previous similar messages [ 3178.708903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3178.713355] Lustre: Skipped 1 previous similar message [ 3178.716539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3178.781778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3179.573497] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3183.601810] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3185.645135] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3185.776568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3187.596727] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3187.712771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3189.220730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.220921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3189.223549] Lustre: Skipped 5 previous similar messages [ 3189.224626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.225429] LustreError: Skipped 1 previous similar message [ 3189.226456] Lustre: Skipped 12 previous similar messages [ 3194.674220] LustreError: 203566:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3194.678645] LustreError: 203566:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3194.781860] Lustre: server umount lustre-OST0000 complete [ 3202.810701] Lustre: server umount lustre-MDT0000 complete [ 3204.577074] LustreError: 201435:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.586111] LustreError: 201435:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 3204.600402] LustreError: 202390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640070 with bad export cookie 15276032256489884116 [ 3204.610115] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3204.817629] Lustre: server umount lustre-MDT0001 complete [ 3206.588633] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:47:53 (1755640073) [ 3214.223684] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3218.528648] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3223.235533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3226.523611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3229.541601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3232.626633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3235.997702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3236.040129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.186364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3236.209300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3236.264283] Lustre: lustre-MDT0000: new disk, initializing [ 3236.314176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3238.225564] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3243.980146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3244.023897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3244.072415] Lustre: 206799:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3244.249533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3246.076490] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3250.068774] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3251.812893] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3255.199797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3255.247158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3257.364769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3258.251063] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3262.371802] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3264.401603] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3264.537692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3266.509788] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3266.644060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3272.677683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3272.678192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.679596] LustreError: Skipped 1 previous similar message [ 3272.680138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3272.686577] Lustre: Skipped 4 previous similar messages [ 3272.687657] Lustre: Skipped 4 previous similar messages [ 3273.712234] LustreError: 209706:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3273.715862] LustreError: 209706:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3273.828351] Lustre: server umount lustre-OST0000 complete [ 3281.711701] Lustre: server umount lustre-MDT0000 complete [ 3282.912946] LustreError: 206811:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.921656] LustreError: 206811:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3283.461129] LustreError: 206791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640149 with bad export cookie 15276032256489884998 [ 3283.466601] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3283.467638] LustreError: 206791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3283.678653] Lustre: server umount lustre-MDT0001 complete [ 3287.439201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.632501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.634129] Lustre: Skipped 5 previous similar messages [ 3288.748421] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3291.292546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3292.687236] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3294.668617] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3295.749365] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3298.052434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3299.822144] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3301.376335] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3321.824754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3324.300336] Lustre: server umount lustre-OST0000 complete [ 3332.206167] Lustre: server umount lustre-MDT0000 complete [ 3333.952340] LustreError: 210567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640200 with bad export cookie 15276032256489885880 [ 3333.958295] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3333.959081] LustreError: 210567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3334.171870] Lustre: server umount lustre-MDT0001 complete [ 3433.277520] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3434.709959] Key type lgssc unregistered [ 3434.860547] LNet: 213895:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3434.864972] LNetError: 213895:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3434.876580] LNet: Removed LNI 192.168.204.146@tcp [ 3435.322202] Key type .llcrypt unregistered [ 3435.324693] Key type ._llcrypt unregistered [ 3445.102215] Key type ._llcrypt registered [ 3445.104133] Key type .llcrypt registered [ 3445.169888] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3452.578956] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3453.136664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3453.175022] alg: No test for adler32 (adler32-zlib) [ 3454.085615] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 3454.208971] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3455.815178] Key type lgssc registered [ 3456.343179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3460.592516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3463.808798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3466.758378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3469.879417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.192896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.224532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3473.232903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.351605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3474.365888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3474.409962] Lustre: lustre-MDT0000: new disk, initializing [ 3474.440462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.447746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3476.238320] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3481.858802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3481.898373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3481.955903] Lustre: 217184:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3482.003699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3482.007161] Lustre: Skipped 1 previous similar message [ 3482.059639] Lustre: lustre-MDT0001: new disk, initializing [ 3482.115525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3482.138214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3482.146177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3483.845783] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3487.856881] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3489.624740] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3492.902840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.937353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3493.094467] Lustre: lustre-OST0000: new disk, initializing [ 3493.098150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3493.151150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.611605] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3499.020701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3499.025046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3499.078055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3499.611040] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3501.488626] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3501.588729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3503.369439] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3503.492404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3509.221768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.222717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.223888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.226051] LustreError: Skipped 1 previous similar message [ 3509.238544] Lustre: Skipped 1 previous similar message [ 3510.447180] LustreError: 220093:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3510.546416] Lustre: server umount lustre-OST0000 complete [ 3512.800494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3512.804818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.812463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3512.815839] Lustre: Skipped 1 previous similar message [ 3514.338303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.338340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.417482] LustreError: 220294:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3518.420991] LustreError: 220294:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3518.456820] Lustre: server umount lustre-MDT0000 complete [ 3520.263935] LustreError: 218917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640386 with bad export cookie 14523550694673694377 [ 3520.269472] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.269721] LustreError: 218917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3520.358710] LustreError: 220496:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3520.363432] LustreError: 220496:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3520.476957] Lustre: server umount lustre-MDT0001 complete [ 3528.886647] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3533.451701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.651627] LustreError: 221663:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.659048] LustreError: 221663:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3533.671761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.319816] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3538.913154] LustreError: 221663:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.247593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3539.425254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3541.126622] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3542.560142] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3545.701213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3548.608869] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3552.855411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.894807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3552.974567] Lustre: lustre-OST0001: new disk, initializing [ 3552.977114] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3553.018421] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3553.020435] Lustre: Skipped 1 previous similar message [ 3555.514571] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3558.906433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3558.910895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3558.934965] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3566.525605] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3571.937791] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3581.784699] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:54:08 (1755640448) [ 3584.780882] LustreError: 225019:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3584.784512] LustreError: 225019:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3584.819672] Lustre: server umount lustre-MDT0000 complete [ 3585.505934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3585.508371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.512359] LustreError: 221662:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.479824] LustreError: 221644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640452 with bad export cookie 14523550694673695259 [ 3586.488344] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3586.682619] Lustre: server umount lustre-MDT0001 complete [ 3603.807156] Lustre: 215401:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755640453/real 1755640453] req@ffff907e071b9f80 x1840922304275968/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755640469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3603.820510] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.831185] Lustre: 215399:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755640453/real 1755640453] req@ffff907e071b8a80 x1840922304276096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755640469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3607.007246] Lustre: 215400:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755640457/real 1755640457] req@ffff907e0ea28700 x1840922304276480/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755640473 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3608.952172] LustreError: 225420:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3608.955929] LustreError: 225420:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3608.980884] Lustre: server umount lustre-OST0000 complete [ 3610.625993] Lustre: server umount lustre-OST0001 complete [ 3613.012904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3616.905230] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3621.376331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3624.442494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3626.890065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3630.019993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.301019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.326013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.433623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.449804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3633.510209] Lustre: lustre-MDT0000: new disk, initializing [ 3633.541477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.549241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3635.323470] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3640.813282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3640.852069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3640.906632] Lustre: 227801:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3641.048808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3642.806595] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3646.773257] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3648.513631] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3651.646330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3651.688787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3651.850928] Lustre: lustre-OST0000: new disk, initializing [ 3651.852781] Lustre: Skipped 1 previous similar message [ 3651.854884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3651.857063] Lustre: Skipped 2 previous similar messages [ 3653.196665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3653.200882] Lustre: Skipped 1 previous similar message [ 3653.206539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3653.256094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3654.439554] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3658.583239] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3660.473317] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3660.609660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3662.618651] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3662.762708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3663.847250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3663.848320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.849399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.849408] Lustre: Skipped 1 previous similar message [ 3663.852080] LustreError: Skipped 1 previous similar message [ 3663.858897] Lustre: Skipped 2 previous similar messages [ 3668.967647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3668.971263] Lustre: Skipped 2 previous similar messages [ 3678.175275] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3678.199213] LustreError: 230709:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3678.202954] LustreError: 230709:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3678.314232] Lustre: server umount lustre-OST0000 complete [ 3682.272132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3682.275108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.279540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3682.281491] Lustre: Skipped 2 previous similar messages [ 3686.441089] Lustre: server umount lustre-MDT0000 complete [ 3688.127831] LustreError: 227793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640554 with bad export cookie 14523550694673697387 [ 3688.133579] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3688.134371] LustreError: 227793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3688.376992] Lustre: server umount lustre-MDT0001 complete [ 3692.666902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.937731] LustreError: 231589:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.985960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.989884] Lustre: Skipped 2 previous similar messages [ 3694.935808] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3698.146736] LustreError: 231589:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.184684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3701.266243] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3703.960895] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3705.663682] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3709.099494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3712.036067] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3714.615977] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3864.035392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.038332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3864.042300] Lustre: Skipped 1 previous similar message [ 3864.047420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3864.052111] Lustre: Skipped 3 previous similar messages [ 3867.762157] LustreError: 233859:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3867.765799] LustreError: 233859:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3867.860252] Lustre: server umount lustre-OST0000 complete [ 3875.742128] Lustre: server umount lustre-MDT0000 complete [ 3877.391828] LustreError: 231569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640743 with bad export cookie 14523550694673698269 [ 3877.395155] LustreError: 231569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3877.395402] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3877.583277] Lustre: server umount lustre-MDT0001 complete [ 3977.988264] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3979.716771] Key type lgssc unregistered [ 3979.897766] LNet: 234824:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3979.904376] LNetError: 234824:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3979.918794] LNet: Removed LNI 192.168.204.146@tcp [ 3980.457173] Key type .llcrypt unregistered [ 3980.459279] Key type ._llcrypt unregistered [ 3992.110358] Key type ._llcrypt registered [ 3992.112284] Key type .llcrypt registered [ 3992.189353] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3999.744578] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4000.151276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4000.192350] alg: No test for adler32 (adler32-zlib) [ 4001.088724] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4001.213993] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4002.823271] Key type lgssc registered [ 4003.486092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4008.177465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4011.489652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4014.450957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4017.670045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4025.039438] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4030.075552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4030.111558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4030.121079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.255259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4031.274357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4031.321376] Lustre: lustre-MDT0000: new disk, initializing [ 4031.360377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.368295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4033.245236] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4038.989819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4039.036351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4039.094538] Lustre: 238816:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4039.145678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4039.149263] Lustre: Skipped 1 previous similar message [ 4039.205545] Lustre: lustre-MDT0001: new disk, initializing [ 4039.270114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4039.292172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4039.299262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4041.220640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4044.115632] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4047.363699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4047.408091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4047.579943] Lustre: lustre-OST0000: new disk, initializing [ 4047.583698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4047.640858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4050.479265] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4052.490807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4052.495128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4052.535475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4055.686998] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4055.727170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4055.818876] Lustre: lustre-OST0001: new disk, initializing [ 4055.822442] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4055.865432] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.364343] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4058.590717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4058.595680] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4058.648057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4064.781230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4071.173897] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4082.648184] LustreError: 242315:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4082.700166] Lustre: server umount lustre-MDT0000 complete [ 4084.198897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.206638] Lustre: Skipped 2 previous similar messages [ 4084.472155] LustreError: 238807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640950 with bad export cookie 10987903113969070375 [ 4084.478041] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.478737] LustreError: 238807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4086.605469] LustreError: 242517:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4086.609171] LustreError: 242517:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4086.720308] Lustre: server umount lustre-MDT0001 complete [ 4098.552153] LustreError: 242718:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4098.556511] LustreError: 242718:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4098.583938] Lustre: server umount lustre-OST0000 complete [ 4110.585413] LustreError: 242919:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4110.589031] LustreError: 242919:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4110.699089] Lustre: server umount lustre-OST0001 complete [ 4117.149572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4121.180703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4121.596635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4125.496324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4129.522625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4129.963248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4137.535197] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4141.444079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.455346] Lustre: lustre-MDT0000: reset Object Index mappings [ 4141.700588] LustreError: 246223:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.713212] LustreError: 246223:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 4141.736847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.365408] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4146.790821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4146.798209] Lustre: lustre-MDT0001: reset Object Index mappings [ 4148.670795] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4149.901619] Lustre: 247320:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4153.039628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4153.234586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.238179] Lustre: Skipped 1 previous similar message [ 4155.614597] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4157.355065] LustreError: 248196:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.696542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4162.334012] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4166.065069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4166.066638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4171.696838] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4173.267558] Lustre: 249159:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4181.477671] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.481814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4181.484488] Lustre: Skipped 4 previous similar messages [ 4181.487843] Lustre: Skipped 2 previous similar messages [ 4185.867355] LustreError: 249262:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4185.871083] LustreError: 249262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4185.909055] Lustre: server umount lustre-MDT0000 complete [ 4186.597875] LustreError: 246222:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4186.606689] LustreError: 246222:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4187.649043] LustreError: 246203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641053 with bad export cookie 10987903113969077151 [ 4187.654499] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4187.655438] LustreError: 246203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4187.849987] Lustre: server umount lustre-MDT0001 complete [ 4199.924626] LustreError: 249665:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4199.928238] LustreError: 249665:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4199.945468] Lustre: server umount lustre-OST0000 complete [ 4211.995519] Lustre: server umount lustre-OST0001 complete [ 4219.987654] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:04:46 (1755641086) [ 4226.944438] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4230.922252] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4235.276559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4238.580086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4241.391627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4244.633990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4247.960894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4248.004184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.149174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4248.170168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4248.221639] Lustre: lustre-MDT0000: new disk, initializing [ 4248.256893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4248.259166] Lustre: Skipped 1 previous similar message [ 4248.266700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4250.166840] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4255.872851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4255.919055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4255.986457] Lustre: 253372:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4256.171083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4257.582618] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4261.705681] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4263.506245] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4266.939194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4266.988273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4267.155553] Lustre: lustre-OST0000: new disk, initializing [ 4267.158154] Lustre: Skipped 1 previous similar message [ 4267.161531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4267.164772] Lustre: Skipped 2 previous similar messages [ 4269.195558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4269.200285] Lustre: Skipped 1 previous similar message [ 4269.204514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4269.261123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4269.917801] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4273.983502] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4275.972898] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4276.106862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4278.105392] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4278.236652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4279.783312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4279.785302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.787849] LustreError: Skipped 1 previous similar message [ 4279.796079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.800835] Lustre: Skipped 2 previous similar messages [ 4284.905395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4284.908910] Lustre: Skipped 1 previous similar message [ 4285.299517] LustreError: 256280:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4285.303582] LustreError: 256280:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4285.422780] Lustre: server umount lustre-OST0000 complete [ 4290.019375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.020969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4290.022832] Lustre: Skipped 2 previous similar messages [ 4290.028583] Lustre: Skipped 1 previous similar message [ 4293.434367] Lustre: server umount lustre-MDT0000 complete [ 4295.105472] LustreError: 254149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641161 with bad export cookie 10987903113969079244 [ 4295.110468] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4295.110658] LustreError: 254149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4295.254493] Lustre: server umount lustre-MDT0001 complete [ 4302.737366] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4307.204448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.441725] LustreError: 257854:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.469536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.473295] Lustre: Skipped 2 previous similar messages [ 4309.115282] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4312.546074] LustreError: 257854:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.112635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4314.898411] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4316.305786] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4319.384642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.063836] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4326.035408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4326.077640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4326.167815] Lustre: lustre-OST0001: new disk, initializing [ 4326.171178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4327.325626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4327.331549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4327.388650] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4328.837639] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4334.226688] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4335.937777] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4343.449105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4343.499650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4347.878450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.885223] Lustre: Skipped 2 previous similar messages [ 4347.886074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4347.890889] Lustre: Skipped 2 previous similar messages [ 4352.082231] LustreError: 261336:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4352.086080] LustreError: 261336:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4352.129439] Lustre: server umount lustre-MDT0000 complete [ 4353.000077] LustreError: 259057:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.008678] LustreError: 259057:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4353.877681] LustreError: 258567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641219 with bad export cookie 10987903113969080126 [ 4353.882328] LustreError: 258567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4353.883661] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.103971] Lustre: server umount lustre-MDT0001 complete [ 4366.343943] Lustre: server umount lustre-OST0000 complete [ 4377.261820] Lustre: server umount lustre-OST0001 complete [ 4384.441114] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4389.086370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4389.312321] LustreError: 263109:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.346430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.349783] Lustre: Skipped 3 previous similar messages [ 4391.173688] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4394.879256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4396.723875] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4398.085533] Lustre: 264205:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4401.249515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4403.995655] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4408.111412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4410.761541] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4414.456249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4414.459441] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4419.991950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4421.645487] Lustre: 266052:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4428.950714] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4428.958396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4433.459865] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 18:08:19 (1755641299) [ 4434.916587] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.921289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4434.923422] Lustre: Skipped 2 previous similar messages [ 4434.926748] Lustre: Skipped 1 previous similar message [ 4449.247257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4449.302797] LustreError: 266739:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4449.306247] LustreError: 266739:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4449.344008] Lustre: server umount lustre-MDT0000 complete [ 4450.272476] LustreError: 263108:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.280286] LustreError: 263108:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 4451.034180] LustreError: 264209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641317 with bad export cookie 10987903113969086405 [ 4451.041145] LustreError: 264209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4451.041303] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4451.226476] Lustre: server umount lustre-MDT0001 complete [ 4463.618985] Lustre: server umount lustre-OST0000 complete [ 4475.682372] Lustre: server umount lustre-OST0001 complete [ 4483.623467] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4487.504938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4491.885177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.082787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4497.973958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4501.104887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4504.725939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4504.765859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4504.907916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4504.927947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4504.984525] Lustre: lustre-MDT0000: new disk, initializing [ 4505.020773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4506.734911] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4512.223453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4512.254193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4512.289027] Lustre: 270173:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4512.399329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4513.997535] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4517.721510] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4519.026183] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4522.278190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4522.318280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4522.496567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.498274] Lustre: Skipped 5 previous similar messages [ 4523.584591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4524.825705] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4528.752401] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4529.726566] Lustre: Mounted lustre-client [ 4530.124624] LustreError: 272661:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff907f0308c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4530.159339] Lustre: Unmounted lustre-client [ 4534.753091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.754074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4534.755887] Lustre: Skipped 2 previous similar messages [ 4534.757240] Lustre: Skipped 10 previous similar messages [ 4540.398961] Lustre: server umount lustre-OST0000 complete [ 4543.456607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4548.210460] Lustre: server umount lustre-MDT0000 complete [ 4549.428642] LustreError: 270164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641415 with bad export cookie 10987903113969093811 [ 4549.434536] LustreError: 270164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4549.435252] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4549.604329] Lustre: server umount lustre-MDT0001 complete [ 4553.978773] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4554.968123] Key type lgssc unregistered [ 4580.767504] LNet: 273934:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4580.771311] LNetError: 273934:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4658.597590] LNet: Removed LNI 192.168.204.146@tcp [ 4659.073161] Key type .llcrypt unregistered [ 4659.075114] Key type ._llcrypt unregistered [ 4667.269718] Key type ._llcrypt registered [ 4667.272535] Key type .llcrypt registered [ 4667.338387] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4675.408171] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4675.746056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4675.772893] alg: No test for adler32 (adler32-zlib) [ 4676.702674] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4676.819581] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4678.439294] Key type lgssc registered [ 4679.012818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4683.667910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4686.903136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4689.844579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4693.101712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4693.997983] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 18:12:40 (1755641560) [ 4718.501679] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 18:13:04 (1755641584) [ 4737.291575] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4738.933899] Key type lgssc unregistered [ 4739.137799] LNet: 279631:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4739.143740] LNetError: 279631:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4739.153549] LNet: Removed LNI 192.168.204.146@tcp [ 4739.630884] Key type .llcrypt unregistered [ 4739.632875] Key type ._llcrypt unregistered [ 4774.568126] Key type ._llcrypt registered [ 4774.569234] Key type .llcrypt registered [ 4774.614463] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4774.752270] Key type .llcrypt unregistered [ 4774.754178] Key type ._llcrypt unregistered [ 4787.837201] Key type ._llcrypt registered [ 4787.838942] Key type .llcrypt registered [ 4787.901605] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4788.067337] Key type .llcrypt unregistered [ 4788.069320] Key type ._llcrypt unregistered [ 4825.445680] Key type ._llcrypt registered [ 4825.447097] Key type .llcrypt registered [ 4825.494190] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4825.635355] Key type .llcrypt unregistered [ 4825.636844] Key type ._llcrypt unregistered [ 4839.696511] Key type ._llcrypt registered [ 4839.697956] Key type .llcrypt registered [ 4839.738494] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4839.883432] Key type .llcrypt unregistered [ 4839.885716] Key type ._llcrypt unregistered [ 4854.439737] Key type ._llcrypt registered [ 4854.441400] Key type .llcrypt registered [ 4854.493836] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4854.646240] Key type .llcrypt unregistered [ 4854.647981] Key type ._llcrypt unregistered [ 4892.208044] Key type ._llcrypt registered [ 4892.209409] Key type .llcrypt registered [ 4892.271720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4892.420424] Key type .llcrypt unregistered [ 4892.422252] Key type ._llcrypt unregistered [ 4907.274435] Key type ._llcrypt registered [ 4907.276354] Key type .llcrypt registered [ 4907.336356] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4907.504704] Key type .llcrypt unregistered [ 4907.506713] Key type ._llcrypt unregistered [ 4922.208805] Key type ._llcrypt registered [ 4922.209671] Key type .llcrypt registered [ 4922.244082] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4922.370491] Key type .llcrypt unregistered [ 4922.371670] Key type ._llcrypt unregistered [ 4949.038825] Key type ._llcrypt registered [ 4949.039783] Key type .llcrypt registered [ 4949.083540] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4956.626131] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4956.942243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4956.976044] alg: No test for adler32 (adler32-zlib) [ 4957.857119] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4957.970718] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4959.551158] Key type lgssc registered [ 4959.954708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.560651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4968.442739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4971.511848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4974.939598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.833877] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4849 sec ========== 18:17:22 (1755641842) [ 4976.717050] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:17:23 (1755641843) === [ 4978.283983] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:17:24 (1755641844) === [ 4996.446213] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4997.923920] Key type lgssc unregistered [ 4998.075674] LNet: 295442:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4998.080458] LNetError: 295442:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4998.094584] LNet: Removed LNI 192.168.204.146@tcp [ 4998.529493] Key type .llcrypt unregistered [ 4998.531666] Key type ._llcrypt unregistered