[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 531474184 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003153] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010163] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.022176] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027253] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030164] *** VALIDATE tmpfs *** [ 0.032471] *** VALIDATE proc *** [ 0.034036] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037051] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038178] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040033] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044104] debug: unmapping init [mem 0xffffffffab859000-0xffffffffab860fff] [ 0.047278] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048810] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049028] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051019] ... generic registers: 4 [ 0.052011] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054018] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056336] rcu: Hierarchical SRCU implementation. [ 0.058781] smp: Bringing up secondary CPUs ... [ 0.059687] x86: Booting SMP configuration: [ 0.060055] .... node #0, CPUs: #1 #2 #3 [ 0.077081] smp: Brought up 1 node, 4 CPUs [ 0.079015] smpboot: Max logical packages: 1 [ 0.080014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.151142] node 0 deferred pages initialised in 64ms [ 0.156036] devtmpfs: initialized [ 0.157398] x86/mm: Memory block size: 128MB [ 0.159791] gcov: version magic: 0x41383552 [ 0.161336] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163408] pinctrl core: initialized pinctrl subsystem [ 0.164214] [ 0.164831] ************************************************************* [ 0.165013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166016] ** ** [ 0.167015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168016] ** ** [ 0.169016] ** This means that this kernel is built to expose internal ** [ 0.170017] ** IOMMU data structures, which may compromise security on ** [ 0.171012] ** your system. ** [ 0.172015] ** ** [ 0.173015] ** If you see this message and you are not debugging the ** [ 0.174015] ** kernel, report this immediately to your vendor! ** [ 0.175013] ** ** [ 0.176015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177025] ************************************************************* [ 0.179054] NET: Registered protocol family 16 [ 0.180617] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182228] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183886] cpuidle: using governor menu [ 0.185990] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188579] PCI: Using configuration type 1 for base access [ 0.190314] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200269] cryptd: max_cpu_qlen set to 1000 [ 0.201238] ACPI: Added _OSI(Module Device) [ 0.203082] ACPI: Added _OSI(Processor Device) [ 0.204044] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209888] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218360] ACPI: Interpreter enabled [ 0.219064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220013] ACPI: Using IOAPIC for interrupt routing [ 0.221182] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233719] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234243] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238446] acpiphp: Slot [2] registered [ 0.239206] acpiphp: Slot [5] registered [ 0.240121] acpiphp: Slot [6] registered [ 0.241200] acpiphp: Slot [7] registered [ 0.242114] acpiphp: Slot [8] registered [ 0.243118] acpiphp: Slot [9] registered [ 0.244120] acpiphp: Slot [10] registered [ 0.245159] acpiphp: Slot [3] registered [ 0.246141] acpiphp: Slot [4] registered [ 0.247117] acpiphp: Slot [11] registered [ 0.248127] acpiphp: Slot [12] registered [ 0.249135] acpiphp: Slot [13] registered [ 0.250083] acpiphp: Slot [14] registered [ 0.251116] acpiphp: Slot [15] registered [ 0.252299] acpiphp: Slot [16] registered [ 0.253126] acpiphp: Slot [17] registered [ 0.254153] acpiphp: Slot [18] registered [ 0.255141] acpiphp: Slot [19] registered [ 0.256077] acpiphp: Slot [20] registered [ 0.257101] acpiphp: Slot [21] registered [ 0.258101] acpiphp: Slot [22] registered [ 0.259151] acpiphp: Slot [23] registered [ 0.260101] acpiphp: Slot [24] registered [ 0.261089] acpiphp: Slot [25] registered [ 0.262103] acpiphp: Slot [26] registered [ 0.263082] acpiphp: Slot [27] registered [ 0.264147] acpiphp: Slot [28] registered [ 0.265087] acpiphp: Slot [29] registered [ 0.266101] acpiphp: Slot [30] registered [ 0.267128] acpiphp: Slot [31] registered [ 0.268058] PCI host bridge to bus 0000:00 [ 0.269020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.270026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.272025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.273028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.274032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.275190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.277274] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.279265] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.285132] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.288126] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.289020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.290028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.291020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.292798] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.293773] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.294044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.295869] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.298017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.307020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.310015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.314115] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.317017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.320020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.327024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.332545] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.339032] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.352026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.375027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.389403] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.402025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.422021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.454017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.473415] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.495024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.538019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.554096] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.562018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.569013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.599029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.612645] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.623016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.632018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.657024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.678871] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.685456] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.687757] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.691499] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.694388] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.700099] iommu: Default domain type: Passthrough [ 0.701499] SCSI subsystem initialized [ 0.703113] ACPI: bus type USB registered [ 0.705152] usbcore: registered new interface driver usbfs [ 0.708185] usbcore: registered new interface driver hub [ 0.711085] usbcore: registered new device driver usb [ 0.714359] pps_core: LinuxPPS API ver. 1 registered [ 0.717009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.722110] PTP clock support registered [ 0.724142] EDAC MC: Ver: 3.0.0 [ 0.726153] PCI: Using ACPI for IRQ routing [ 0.727954] NetLabel: Initializing [ 0.730013] NetLabel: domain hash size = 128 [ 0.732010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.735119] NetLabel: unlabeled traffic allowed by default [ 0.738483] vgaarb: loaded [ 0.740454] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748823] clocksource: Switched to clocksource kvm-clock [ 0.887045] VFS: Disk quotas dquot_6.6.0 [ 0.889177] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.893119] *** VALIDATE ramfs *** [ 0.895078] *** VALIDATE hugetlbfs *** [ 0.897986] pnp: PnP ACPI init [ 0.902555] pnp: PnP ACPI: found 6 devices [ 0.922141] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.926128] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.929046] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.932116] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.935171] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.938031] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.941187] NET: Registered protocol family 2 [ 0.945175] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.952772] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.959582] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.964505] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.967874] TCP: Hash tables configured (established 65536 bind 65536) [ 0.970754] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.973849] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.976959] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.980569] NET: Registered protocol family 1 [ 0.982684] RPC: Registered named UNIX socket transport module. [ 0.984794] RPC: Registered udp transport module. [ 0.986203] RPC: Registered tcp transport module. [ 0.987680] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.989986] NET: Registered protocol family 44 [ 0.991522] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.993412] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.995300] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.997895] PCI: CLS 0 bytes, default 64 [ 0.999578] Unpacking initramfs... [ 2.836897] debug: unmapping init [mem 0xffff8c2bfcc54000-0xffff8c2bfffbffff] [ 2.842725] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.847353] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.851651] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.767864] Initialise system trusted keyrings [ 3.769139] Key type blacklist registered [ 3.773793] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.793211] zbud: loaded [ 3.803224] *** VALIDATE nfs *** [ 3.804754] *** VALIDATE nfs4 *** [ 3.809492] pstore: using deflate compression [ 3.841852] Platform Keyring initialized [ 4.175912] NET: Registered protocol family 38 [ 4.177927] Key type asymmetric registered [ 4.184327] Asymmetric key parser 'x509' registered [ 4.190378] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.197576] io scheduler mq-deadline registered [ 4.200894] io scheduler kyber registered [ 4.205478] io scheduler bfq registered [ 4.209596] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.214250] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.221760] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.229864] ACPI: Power Button [PWRF] [ 4.243642] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.259268] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.282084] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.301791] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.344864] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.387408] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.445623] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.463212] Non-volatile memory driver v1.3 [ 4.469504] Linux agpgart interface v0.103 [ 4.546964] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.555695] vda: detected capacity change from 0 to 74694656 [ 4.577441] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.583335] vdb: detected capacity change from 0 to 1073741824 [ 4.606468] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.609537] vdc: detected capacity change from 0 to 2621440000 [ 4.638617] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.642663] vdd: detected capacity change from 0 to 2621440000 [ 4.673457] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.678869] vde: detected capacity change from 0 to 4294967296 [ 4.715782] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.720122] vdf: detected capacity change from 0 to 4294967296 [ 4.732723] libphy: Fixed MDIO Bus: probed [ 4.738740] usbcore: registered new interface driver usbserial_generic [ 4.742444] usbserial: USB Serial support registered for generic [ 4.746485] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.751816] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.754747] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.757336] mousedev: PS/2 mouse device common for all mice [ 4.760811] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.765601] rtc_cmos 00:05: RTC can wake from S4 [ 4.767840] rtc_cmos 00:05: registered as rtc0 [ 4.767902] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.767937] intel_pstate: CPU model not supported [ 4.772892] hid: raw HID events driver (C) Jiri Kosina [ 4.774216] usbcore: registered new interface driver usbhid [ 4.791668] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.793206] usbhid: USB HID core driver [ 4.799622] drop_monitor: Initializing network drop monitor service [ 4.803895] Initializing XFRM netlink socket [ 4.805784] NET: Registered protocol family 10 [ 4.807539] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.812477] Segment Routing with IPv6 [ 4.818050] NET: Registered protocol family 17 [ 4.821118] mpls_gso: MPLS GSO support [ 4.827471] RAS: Correctable Errors collector initialized. [ 4.829608] AVX version of gcm_enc/dec engaged. [ 4.831304] AES CTR mode by8 optimization enabled [ 4.923326] sched_clock: Marking stable (4923304922, 0)->(6085103383, -1161798461) [ 4.927653] registered taskstats version 1 [ 4.929551] Loading compiled-in X.509 certificates [ 4.932359] zswap: loaded using pool lzo/zbud [ 4.963696] Key type big_key registered [ 4.981451] Key type encrypted registered [ 4.982954] ima: No TPM chip found, activating TPM-bypass! [ 4.985871] ima: Allocated hash algorithm: sha1 [ 4.987651] ima: No architecture policies found [ 4.989561] evm: Initialising EVM extended attributes: [ 4.991847] evm: security.selinux [ 4.993303] evm: security.ima [ 4.994421] evm: security.capability [ 4.995836] evm: HMAC attrs: 0x1 [ 4.998339] rtc_cmos 00:05: setting system clock to 2026-08-11 16:01:30 UTC (1786464090) [ 5.004862] debug: unmapping init [mem 0xffffffffac803000-0xffffffffac9fffff] [ 5.008441] debug: unmapping init [mem 0xffffffffab582000-0xffffffffab858fff] [ 5.017190] Write protecting the kernel read-only data: 28672k [ 5.023901] debug: unmapping init [mem 0xffffffffa9c03000-0xffffffffa9dfffff] [ 5.026956] debug: unmapping init [mem 0xffffffffaa514000-0xffffffffaa5fffff] [ 5.075906] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.088517] systemd[1]: Detected virtualization kvm. [ 5.091694] systemd[1]: Detected architecture x86-64. [ 5.094120] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.126929] systemd[1]: No hostname configured. [ 5.129024] systemd[1]: Set hostname to . [ 5.131462] random: systemd: uninitialized urandom read (16 bytes read) [ 5.134691] systemd[1]: Initializing machine ID from random generator. [ 5.280633] random: ln: uninitialized urandom read (6 bytes read) [ 5.511778] random: systemd: uninitialized urandom read (16 bytes read) [ 5.516883] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.523838] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.534331] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.758779] device-mapper: uevent: version 1.0.3 [ 6.760891] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.209416] random: fast init done [ 8.247143] virtio_net virtio0 ens2: renamed from eth0 [ 8.702551] scsi host0: ata_piix [ 8.782479] scsi host1: ata_piix [ 8.786478] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.791920] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.528628] random: crng init done [ 13.530730] random: 7 urandom warning(s) missed due to ratelimiting [ 16.621486] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.041753] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.049130] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.679292] SELinux: Disabled at runtime. [ 20.792540] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.806766] systemd[1]: Detected virtualization kvm. [ 20.810967] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.966146] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.975090] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.996618] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.006400] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.014170] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.028342] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.039352] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 22.289519] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 23.359057] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.409441] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.464539] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.951358] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.022644] EDAC sbridge: Ver: 1.1.2 [ 28.899139] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.338394] NFS: Registering the id_resolver key type [ 29.340102] Key type id_resolver registered [ 29.341230] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 59.081698] libcfs: loading out-of-tree module taints kernel. [ 59.110059] Key type ._llcrypt registered [ 59.112265] Key type .llcrypt registered [ 59.167391] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 67.516816] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 68.160626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.169395] alg: No test for adler32 (adler32-zlib) [ 69.190273] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 69.536825] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 71.167503] Key type lgssc registered [ 71.851778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.494419] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.350435] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 102.889071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 102.900591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 104.012257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 104.026200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 104.068504] Lustre: lustre-MDT0000: new disk, initializing [ 104.116670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 104.129192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 105.775694] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.891817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 111.940133] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 111.954450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 111.956143] Lustre: Skipped 1 previous similar message [ 112.004669] Lustre: lustre-MDT0001: new disk, initializing [ 112.032220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 112.044092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 112.048758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 113.740976] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.432801] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 120.633918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 120.788534] Lustre: lustre-OST0000: new disk, initializing [ 120.790717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 120.793816] Lustre: 8414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 120.832351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 123.430398] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.430269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 125.433309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 125.465106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 129.929947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 129.997563] Lustre: lustre-OST0001: new disk, initializing [ 130.000179] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 130.003807] Lustre: 9485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 130.035362] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 130.868873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 130.873154] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 130.914793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 132.661643] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 139.701606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 146.740432] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 149.609457] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 151.590968] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 153.531273] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 154.671730] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 155.705773] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 156.434368] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 11 12:04:00 EDT 2026 [ 163.935428] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 164.657077] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 177.120252] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 177.121211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 177.127621] Lustre: Skipped 3 previous similar messages [ 177.130529] Lustre: Skipped 2 previous similar messages [ 182.240177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 182.244049] Lustre: Skipped 3 previous similar messages [ 182.830742] Lustre: server umount lustre-MDT0000 complete [ 184.459688] LustreError: 10301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464269 with bad export cookie 5583290871191453424 [ 184.461215] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 184.465508] LustreError: 10301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 187.360179] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 187.361079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 187.366561] Lustre: Skipped 1 previous similar message [ 189.407858] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 189.412233] Lustre: Skipped 2 previous similar messages [ 190.656254] Lustre: server umount lustre-MDT0001 complete [ 192.281603] Lustre: server umount lustre-OST0000 complete [ 193.933189] Lustre: server umount lustre-OST0001 complete [ 196.528390] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 200.560974] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 216.123180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.221578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.235297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.275303] Lustre: lustre-MDT0000: new disk, initializing [ 216.308549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.320079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 218.192514] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 223.339347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.395897] Lustre: 15024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.545162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.216314] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 229.817280] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 232.199224] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 235.039978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.170386] Lustre: lustre-OST0000: new disk, initializing [ 235.172433] Lustre: Skipped 1 previous similar message [ 235.174528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.177300] Lustre: Skipped 2 previous similar messages [ 235.181559] Lustre: 16995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.224366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.228267] Lustre: Skipped 1 previous similar message [ 237.005075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 237.008600] Lustre: Skipped 1 previous similar message [ 237.011199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 237.062319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.605682] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 242.318636] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 244.114081] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 244.211315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 246.041420] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 246.135956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 247.265520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 247.266705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 247.268710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 247.268821] LustreError: Skipped 1 previous similar message [ 247.273673] Lustre: Skipped 1 previous similar message [ 253.146979] Lustre: server umount lustre-OST0000 complete [ 257.504925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.505913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 257.509949] Lustre: Skipped 1 previous similar message [ 257.512304] Lustre: Skipped 3 previous similar messages [ 260.704616] Lustre: server umount lustre-MDT0000 complete [ 262.185583] LustreError: 15017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464347 with bad export cookie 5583290871191454957 [ 262.187533] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 262.188850] LustreError: 15017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 262.334460] Lustre: server umount lustre-MDT0001 complete [ 264.737527] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:05:49 (1786464349) [ 272.116096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.287464] LustreError: 19864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.294885] LustreError: 19864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 272.315149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.911335] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 277.286975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.012899] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 282.007708] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 284.220316] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 285.058793] Lustre: Failing over lustre-MDT0000 [ 285.163801] Lustre: server umount lustre-MDT0000 complete [ 286.688544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.689240] LustreError: 19889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.692333] Lustre: Skipped 1 previous similar message [ 287.712499] LustreError: 19864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.720887] LustreError: 19864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 292.832171] LustreError: 19865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.839911] LustreError: 19865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 297.953460] LustreError: 20592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.964361] LustreError: 20592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 298.697958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 298.753329] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.351262] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 302.946593] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 302.949736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 302.958840] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 304.744789] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 308.193126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 308.193433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.194322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.203124] Lustre: Skipped 1 previous similar message [ 312.990158] Lustre: server umount lustre-MDT0000 complete [ 313.312433] LustreError: 19859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.318462] LustreError: 19859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 314.493438] LustreError: 19845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464399 with bad export cookie 5583290871191456371 [ 314.497506] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.498057] LustreError: 19845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 314.626810] Lustre: server umount lustre-MDT0001 complete [ 317.914521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.153940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 318.156489] Lustre: Skipped 2 previous similar messages [ 319.876961] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 323.794340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 325.724878] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 328.611747] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 330.786840] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 331.651151] Lustre: Failing over lustre-MDT0000 [ 331.747841] Lustre: server umount lustre-MDT0000 complete [ 333.280511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.281804] LustreError: 23192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.285756] Lustre: Skipped 1 previous similar message [ 333.293635] LustreError: 23192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 345.838554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.902093] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.851681] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 349.091439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 349.098152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 349.101459] Lustre: Skipped 1 previous similar message [ 349.107806] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 352.224636] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 359.392622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 359.393575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.398724] Lustre: Skipped 2 previous similar messages [ 360.610526] Lustre: server umount lustre-MDT0000 complete [ 362.116655] LustreError: 23177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464447 with bad export cookie 5583290871191457309 [ 362.119796] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.123570] LustreError: 23177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 362.272509] Lustre: server umount lustre-MDT0001 complete [ 365.808467] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:07:30 (1786464450) [ 372.136867] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 375.959617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 391.925701] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 396.594089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.696319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.714641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.753343] Lustre: lustre-MDT0000: new disk, initializing [ 396.791190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.800035] Lustre: Skipped 2 previous similar messages [ 396.809722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 398.534870] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 403.597527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.654641] Lustre: 29574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 403.778735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 405.413198] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 409.810578] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 412.067779] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 414.780409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.904121] Lustre: 31538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 416.851213] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 416.879646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 416.886486] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 417.173367] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 421.590544] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 424.130316] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 432.096226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 432.099554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.104880] Lustre: Skipped 2 previous similar messages [ 432.107801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.565607] Lustre: server umount lustre-OST0000 complete [ 443.177679] Lustre: server umount lustre-MDT0000 complete [ 444.604454] LustreError: 30368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464530 with bad export cookie 5583290871191457715 [ 444.606816] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 444.609297] LustreError: 30368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 444.788664] Lustre: server umount lustre-MDT0001 complete [ 448.868515] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 450.055455] Key type lgssc unregistered [ 450.194459] LNet: 33492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.198079] LNetError: 33492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.207443] LNet: Removed LNI 192.168.204.146@tcp [ 450.650111] Key type .llcrypt unregistered [ 450.652168] Key type ._llcrypt unregistered [ 460.862492] Key type ._llcrypt registered [ 460.863572] Key type .llcrypt registered [ 460.907786] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 461.345788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 461.353651] alg: No test for adler32 (adler32-zlib) [ 462.220357] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 462.319352] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 463.903150] Key type lgssc registered [ 464.338565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 467.320506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 467.327425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.493568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.014645] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 473.290771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.447638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 474.981950] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 477.775730] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 479.804167] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 482.290184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.441371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.495064] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 484.589663] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 487.297768] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 499.320860] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:09:43 (1786464583) [ 500.060137] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 500.762323] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 503.323287] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:09:47 (1786464587) [ 506.436273] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:09:51 (1786464591) [ 506.842976] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 510.264140] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 510.268537] Lustre: Skipped 2 previous similar messages [ 511.023206] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:09:55 (1786464595) [ 514.530782] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 514.534556] Lustre: Skipped 4 previous similar messages [ 520.962769] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:10:05 (1786464605) [ 522.186526] Lustre: *** cfs_fail_loc=131b, val=2*** [ 523.680895] Lustre: *** cfs_fail_loc=131b, val=2*** [ 527.017295] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:10:11 (1786464611) [ 527.786836] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 527.789301] Lustre: Skipped 9 previous similar messages [ 532.669453] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:10:17 (1786464617) [ 539.105041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.105929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.108696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.112537] Lustre: Skipped 1 previous similar message [ 539.116469] Lustre: Skipped 1 previous similar message [ 544.224717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.345279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.347731] Lustre: Skipped 2 previous similar messages [ 551.392201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 551.464657] Lustre: server umount lustre-OST0000 complete [ 554.464662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.465612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.469721] Lustre: Skipped 1 previous similar message [ 554.473380] Lustre: Skipped 1 previous similar message [ 559.586379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.592843] Lustre: Skipped 1 previous similar message [ 567.263215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 567.377918] Lustre: server umount lustre-MDT0000 complete [ 569.829938] LustreError: 34899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.849985] LustreError: 34899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 570.839165] LustreError: 37797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464656 with bad export cookie 17870770759662299338 [ 570.854653] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.858664] LustreError: 37797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 571.195353] Lustre: server umount lustre-MDT0001 complete [ 584.255631] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 586.529263] Key type lgssc unregistered [ 586.751115] LNet: 42720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.759797] LNetError: 42720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.779724] LNet: Removed LNI 192.168.204.146@tcp [ 587.444313] Key type .llcrypt unregistered [ 587.448065] Key type ._llcrypt unregistered [ 605.493362] Key type ._llcrypt registered [ 605.495073] Key type .llcrypt registered [ 605.602569] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 619.609169] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 620.223751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.282598] alg: No test for adler32 (adler32-zlib) [ 621.208228] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 621.348118] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 622.951200] Key type lgssc registered [ 623.553948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 641.461878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 641.473292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.597156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 642.616858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 642.666634] Lustre: lustre-MDT0000: new disk, initializing [ 642.716425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.732475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 644.865248] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 651.059044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.124947] Lustre: 45869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 651.165907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 651.168460] Lustre: Skipped 1 previous similar message [ 651.222493] Lustre: lustre-MDT0001: new disk, initializing [ 651.289402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 651.318216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 651.325490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 653.739280] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 659.135098] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 661.997265] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.400488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 665.558298] Lustre: lustre-OST0000: new disk, initializing [ 665.561400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 665.565803] Lustre: 47840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 665.618484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.637262] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 672.265958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 672.271163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 672.339614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 674.237266] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 676.758636] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 676.877829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 679.148182] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 679.276033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 682.467732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 682.467917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.469733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 682.473327] LustreError: Skipped 1 previous similar message [ 682.485131] Lustre: Skipped 1 previous similar message [ 686.449723] Lustre: server umount lustre-OST0000 complete [ 692.192045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 692.197817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.206116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 692.210028] Lustre: Skipped 1 previous similar message [ 694.640281] Lustre: server umount lustre-MDT0000 complete [ 696.485367] LustreError: 45862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464781 with bad export cookie 6253713537187077474 [ 696.491254] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.493447] LustreError: 45862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 696.673601] Lustre: server umount lustre-MDT0001 complete [ 701.903098] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:13:06 (1786464786) [ 705.397487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.666862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.828567] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 711.136754] LustreError: 50296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.143911] LustreError: 50296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 712.376214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.573199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 714.515629] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 718.451293] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 721.508791] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 725.501119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 725.745468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.173908] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 733.115873] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 740.094452] LustreError: 52818:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 740.617141] Lustre: Modifying parameter general.timeout=1 in log params [ 742.074385] Lustre: Modifying parameter general.timeout=4 in log params [ 742.077984] Lustre: Skipped 2 previous similar messages [ 744.198676] Lustre: Modifying parameter general.timeout=8 in log params [ 744.201760] Lustre: Skipped 3 previous similar messages [ 748.235467] Lustre: Modifying parameter general.timeout=16 in log params [ 748.238678] Lustre: Skipped 7 previous similar messages [ 756.624214] Lustre: Modifying parameter general.timeout=33 in log params [ 756.626374] Lustre: Skipped 16 previous similar messages [ 772.770657] Lustre: Modifying parameter general.timeout=67 in log params [ 772.773608] Lustre: Skipped 33 previous similar messages [ 804.914117] Lustre: Disabling parameter general.timeout= in log params [ 804.915890] Lustre: Skipped 62 previous similar messages [ 806.454520] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:14:50 (1786464890) [ 807.752057] Lustre: 59730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 809.285082] Lustre: 59927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 809.290399] Lustre: 59927:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 811.568059] Lustre: 60221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 811.574046] Lustre: 60221:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 823.029358] Lustre: 61564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 823.034370] LustreError: 61564:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 824.755546] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:15:09 (1786464909) [ 829.408989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.410716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 829.414393] Lustre: Skipped 3 previous similar messages [ 829.417179] Lustre: Skipped 2 previous similar messages [ 834.528838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 834.532218] Lustre: Skipped 2 previous similar messages [ 835.557173] Lustre: server umount lustre-MDT0000 complete [ 837.041150] LustreError: 52134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464922 with bad export cookie 6253713537187078356 [ 837.044062] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.045566] LustreError: 52134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 837.198276] Lustre: server umount lustre-MDT0001 complete [ 849.189348] Lustre: server umount lustre-OST0000 complete [ 859.640398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.754111] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 859.863577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.553856] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 865.125038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.172944] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 865.185706] Lustre: 64089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 866.964306] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 870.142822] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 872.454104] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 875.238641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.324084] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 877.930111] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 881.090743] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 881.764989] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 881.772533] Lustre: Skipped 23 previous similar messages [ 881.801681] Lustre: 66655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 886.242069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.242270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.245553] LustreError: Skipped 1 previous similar message [ 886.251300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.254776] Lustre: Skipped 1 previous similar message [ 890.191374] Lustre: server umount lustre-OST0000 complete [ 895.968083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 895.970704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.976421] Lustre: Skipped 1 previous similar message [ 895.979408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.630587] Lustre: server umount lustre-MDT0000 complete [ 899.144123] LustreError: 64081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464984 with bad export cookie 6253713537187080862 [ 899.146980] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 899.150017] LustreError: 64081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 899.274571] Lustre: server umount lustre-MDT0001 complete [ 903.577710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 904.745426] Key type lgssc unregistered [ 904.885453] LNet: 67864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.888196] LNetError: 67864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.894383] LNet: Removed LNI 192.168.204.146@tcp [ 905.239350] Key type .llcrypt unregistered [ 905.241078] Key type ._llcrypt unregistered [ 915.479362] Key type ._llcrypt registered [ 915.480756] Key type .llcrypt registered [ 915.525412] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 915.954943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.985684] alg: No test for adler32 (adler32-zlib) [ 916.855228] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 916.958856] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 918.551163] Key type lgssc registered [ 918.992722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.135559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.143874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.353344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.967434] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 928.373319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.624188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 930.153637] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 933.011151] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 935.045642] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 937.535439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 937.752429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.000976] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 942.994636] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 951.065846] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 954.336329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 954.339577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.344857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.432791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.434393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.439311] Lustre: Skipped 1 previous similar message [ 959.775062] Lustre: server umount lustre-MDT0000 complete [ 961.063993] LustreError: 69233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465046 with bad export cookie 17642101327628414705 [ 961.065926] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.068113] LustreError: 69233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 961.075152] LustreError: 69245:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 961.172936] Lustre: server umount lustre-MDT0001 complete [ 987.208695] Lustre: server umount lustre-OST0000 complete [ 996.680306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.784157] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 996.796619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 996.883274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.276138] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1001.208824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.247043] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1001.257992] Lustre: 74121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1001.424797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1002.844245] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1005.275236] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1007.213159] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1009.733310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1009.844272] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1010.028885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.424864] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1015.547960] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.299716] Lustre: 76695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1017.084246] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1022.947973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1022.948959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.952747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.956337] Lustre: Skipped 1 previous similar message [ 1022.958779] Lustre: Skipped 2 previous similar messages [ 1024.728886] Lustre: server umount lustre-OST0000 complete [ 1027.040363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1027.043699] LustreError: Skipped 1 previous similar message [ 1027.045632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.051346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1027.054204] Lustre: Skipped 1 previous similar message [ 1033.184283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.187354] Lustre: Skipped 3 previous similar messages [ 1040.351143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1040.419048] Lustre: server umount lustre-MDT0000 complete [ 1041.885270] LustreError: 76697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465127 with bad export cookie 17642101327628415503 [ 1041.887660] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1041.890656] LustreError: 76697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1042.022022] Lustre: server umount lustre-MDT0001 complete [ 1046.273162] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1047.416460] Key type lgssc unregistered [ 1047.547343] LNet: 77904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.550372] LNetError: 77904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.562361] LNet: Removed LNI 192.168.204.146@tcp [ 1047.905223] Key type .llcrypt unregistered [ 1047.906663] Key type ._llcrypt unregistered [ 1057.494073] Key type ._llcrypt registered [ 1057.495263] Key type .llcrypt registered [ 1057.536845] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1057.873834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.901028] alg: No test for adler32 (adler32-zlib) [ 1058.788575] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1058.889122] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1060.471165] Key type lgssc registered [ 1060.913846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1063.744866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1063.753493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.915547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.376432] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1069.392879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.607720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.029351] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1073.676541] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1075.683664] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.287259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.499065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.664814] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1083.227434] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1095.984078] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1096.569384] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1096.572287] Lustre: Skipped 1 previous similar message [ 1097.766928] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1097.769094] Lustre: Skipped 3 previous similar messages [ 1099.888791] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1099.891330] Lustre: Skipped 6 previous similar messages [ 1103.931948] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1103.934861] Lustre: Skipped 12 previous similar messages [ 1112.107849] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1112.110509] Lustre: Skipped 26 previous similar messages [ 1130.082786] Lustre: 87378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1130.487804] Lustre: Disabling parameter general.jobid_var= in log params [ 1130.491419] Lustre: Skipped 46 previous similar messages [ 1131.154496] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:20:15 (1786465215) [ 1131.738492] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1132.428919] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:20:17 (1786465217) [ 1136.096287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.098930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.102870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.120157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1137.123243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.123767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.131977] Lustre: Skipped 1 previous similar message [ 1139.604603] Lustre: server umount lustre-OST0000 complete [ 1141.215863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1141.218045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.222948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.080427] LustreError: 87874:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000009b39ec3a) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1147.360076] LustreError: 79293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.366032] LustreError: 79293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1149.112249] Lustre: server umount lustre-MDT0000 complete [ 1150.657312] LustreError: 79275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465236 with bad export cookie 11528068470194486546 [ 1150.660136] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.662539] LustreError: 79275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1150.799595] Lustre: server umount lustre-MDT0001 complete [ 1154.769893] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1155.921620] Key type lgssc unregistered [ 1156.063574] LNet: 88628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.067113] LNetError: 88628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.074486] LNet: Removed LNI 192.168.204.146@tcp [ 1156.419135] Key type .llcrypt unregistered [ 1156.420805] Key type ._llcrypt unregistered [ 1157.971333] Key type ._llcrypt registered [ 1157.973414] Key type .llcrypt registered [ 1158.017728] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1159.520690] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules [ 1159.917526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.955895] alg: No test for adler32 (adler32-zlib) [ 1160.817234] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1160.905536] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1162.487140] Key type lgssc registered [ 1162.928687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1163.352913] LustreError: 89782:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1171.297341] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1174.405871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1184.542525] LustreError: 89782:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1184.556385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.734797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.166960] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1190.080380] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:21:14 (1786465274) [ 1191.651244] LustreError: 90836:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1191.654741] LustreError: 90836:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1191.656943] LustreError: 90836:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1191.774218] Lustre: server umount lustre-MDT0000 complete [ 1196.238941] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1197.229644] Key type lgssc unregistered [ 1197.354383] LNet: 92329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.356589] LNetError: 92329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.362349] LNet: Removed LNI 192.168.204.146@tcp [ 1197.659843] Key type .llcrypt unregistered [ 1197.661223] Key type ._llcrypt unregistered [ 1204.969749] Key type ._llcrypt registered [ 1204.970837] Key type .llcrypt registered [ 1205.008886] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1205.442806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1205.453200] alg: No test for adler32 (adler32-zlib) [ 1206.319252] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1206.418734] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1208.007127] Key type lgssc registered [ 1208.396387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.199728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1211.210761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.385846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.776283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1216.762923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.024112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1218.508789] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1221.019618] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.903301] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.274418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.480716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.444914] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1230.002792] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1260.899464] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:22:25 (1786465345) [ 1261.450357] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1261.999186] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:22:26 (1786465346) [ 1263.071591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1263.073600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.077502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.585495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.585920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.588776] Lustre: Skipped 1 previous similar message [ 1263.591618] Lustre: Skipped 1 previous similar message [ 1268.704794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.706926] Lustre: Skipped 2 previous similar messages [ 1269.076587] Lustre: server umount lustre-MDT0000 complete [ 1270.262535] LustreError: 93505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465355 with bad export cookie 9423233071706879870 [ 1270.264875] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1270.266030] LustreError: 93505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1270.373339] Lustre: server umount lustre-MDT0001 complete [ 1281.909407] Lustre: server umount lustre-OST0000 complete [ 1285.702523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.851736] LustreError: 97814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.858937] LustreError: 97814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1285.907603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.265416] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1289.954425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.177296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1291.478945] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1293.773337] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1295.520161] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1299.421368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.490537] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1299.492992] LustreError: 97804:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1299.505851] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1299.509649] LustreError: 99896:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1299.511632] LustreError: 99896:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1299.515080] LustreError: 99896:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1299.543014] Lustre: server umount lustre-OST0000 complete [ 1299.544730] LustreError: 99896:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1302.113467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.171588] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1302.173995] LustreError: 97804:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1302.182521] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1302.184890] LustreError: 100363:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1302.187209] LustreError: 100363:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1302.189075] LustreError: 100363:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1302.208132] Lustre: server umount lustre-OST0000 complete [ 1302.209324] LustreError: 100363:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1305.153589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1305.216385] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1305.217947] Lustre: Found index 0 for lustre-OST0000, updating log [ 1305.219956] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1305.239668] Lustre: lustre-OST0000: new disk, initializing [ 1305.241096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1305.243487] Lustre: 100892:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1305.328528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.148445] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1308.283163] Lustre: Failing over lustre-OST0000 [ 1308.332848] Lustre: server umount lustre-OST0000 complete [ 1310.688320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.688383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1310.688892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.692909] Lustre: Skipped 1 previous similar message [ 1314.784173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.786277] Lustre: Skipped 2 previous similar messages [ 1315.666942] Lustre: server umount lustre-MDT0000 complete [ 1316.837942] LustreError: 97796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465402 with bad export cookie 9423233071706882600 [ 1316.839752] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.841475] LustreError: 97796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1316.947129] Lustre: server umount lustre-MDT0001 complete [ 1319.527559] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:23:24 (1786465404) [ 1324.460380] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1327.717280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.831956] LustreError: 103227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.857782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.140178] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1331.900670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.424211] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1334.346630] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1336.442476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1338.571809] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1341.335128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1341.365627] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1341.397943] Lustre: lustre-OST0001: new disk, initializing [ 1341.399485] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1341.401844] Lustre: 105545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1343.378455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1346.537130] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1347.078549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1347.082671] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1347.124437] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1353.217869] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1354.520095] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 1357.282919] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.283609] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1360.596098] Lustre: server umount lustre-MDT0001 complete [ 1362.402200] LustreError: 103228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.408472] LustreError: 103228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1363.595309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.629223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1363.632097] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1363.647531] Lustre: 103216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1363.882960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1363.886171] Lustre: Skipped 3 previous similar messages [ 1365.229572] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1365.669297] LustreError: 107372:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1365.671788] LustreError: 107372:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1365.674388] LustreError: 103220:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1365.701853] LustreError: 107379:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -17 [ 1365.704204] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1365.704204] [ 1368.182126] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 12:24:12 (1786465452) [ 1369.062361] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1373.311802] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1379.110396] Lustre: 109149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1386.653480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1387.770815] Lustre: 110280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1435.615657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1435.616054] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.616290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.616294] Lustre: Skipped 4 previous similar messages [ 1435.627121] Lustre: Skipped 5 previous similar messages [ 1437.272153] Lustre: server umount lustre-MDT0000 complete [ 1438.463673] LustreError: 104368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465523 with bad export cookie 9423233071706883468 [ 1438.465326] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.468150] LustreError: 104368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1438.573030] Lustre: server umount lustre-MDT0001 complete [ 1449.992132] Lustre: server umount lustre-OST0000 complete [ 1461.391921] Lustre: server umount lustre-OST0001 complete [ 1463.125637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1466.002640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1468.684795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1471.672276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1477.305598] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1480.494116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.503293] Lustre: lustre-MDT0000: reset Object Index mappings [ 1480.626304] LustreError: 114202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.647762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.845885] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1484.464217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.470702] Lustre: lustre-MDT0001: reset Object Index mappings [ 1485.918496] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1486.809794] Lustre: 115349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1488.777068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1490.698666] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1491.047174] LustreError: 115702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.053692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 1493.333914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1494.570692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 1494.571734] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1494.572779] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 1495.360751] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1498.384883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1499.500490] Lustre: 117223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1512.605985] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 12:26:37 (1786465597) [ 1545.699406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.700554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1545.704666] Lustre: Skipped 1 previous similar message [ 1545.707088] Lustre: Skipped 6 previous similar messages [ 1548.325683] Lustre: server umount lustre-MDT0000 complete [ 1549.500483] LustreError: 117226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465635 with bad export cookie 9423233071706930284 [ 1549.504759] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1549.505264] LustreError: 117226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1550.816626] LustreError: 116695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.821534] LustreError: 116695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1555.899576] Lustre: server umount lustre-MDT0001 complete [ 1563.391822] Lustre: server umount lustre-OST0000 complete [ 1570.714797] Lustre: server umount lustre-OST0001 complete [ 1572.815867] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1578.826154] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1612.008615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.182944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1612.201245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1612.253462] Lustre: lustre-MDT0000: new disk, initializing [ 1612.323541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.328549] Lustre: Skipped 3 previous similar messages [ 1612.342567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1617.245701] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1623.539333] Lustre: server umount lustre-MDT0000 complete [ 1630.451217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.664340] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1630.672830] Lustre: 121483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1630.683952] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1632.901385] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1635.448771] Lustre: server umount lustre-MDT0000 complete [ 1640.040300] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 12:28:44 (1786465724) [ 1640.891334] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1641.816033] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 12:28:46 (1786465726) [ 1650.567981] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1655.281584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.307077] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1661.217557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.280638] Lustre: 123532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1661.314529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1661.318024] Lustre: Skipped 1 previous similar message [ 1661.361732] Lustre: lustre-MDT0001: new disk, initializing [ 1661.433728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1661.442148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1663.243606] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1666.135600] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1669.320964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1669.461622] Lustre: lustre-OST0000: new disk, initializing [ 1669.464237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1669.468942] Lustre: 125109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1670.566355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1670.572732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1670.620976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1672.355786] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1678.194159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1678.291266] Lustre: lustre-OST0001: new disk, initializing [ 1678.296750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1678.305807] Lustre: 125979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1679.570962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1679.575053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1679.634125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1681.217940] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1687.000238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1688.739617] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1718.051161] hrtimer: interrupt took 5208897 ns [ 2040.979311] Lustre: 132384:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2040.985950] Lustre: 132384:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2040.991158] Lustre: 132384:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2040.998727] Lustre: 132384:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2041.004287] Lustre: 132384:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2041.009056] Lustre: 132384:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2308.600249] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:39:51 (1786466391) [ 2324.901571] Lustre: server umount lustre-MDT0000 complete [ 2324.963820] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.971293] LustreError: 123539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.975141] Lustre: Skipped 7 previous similar messages [ 2324.998584] LustreError: 123539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2327.752070] LustreError: 123526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786466413 with bad export cookie 9423233071706982385 [ 2327.753106] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.760491] LustreError: 123526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2330.080713] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.081043] LustreError: 126795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.085674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2330.085685] Lustre: Skipped 4 previous similar messages [ 2330.087762] Lustre: Skipped 1 previous similar message [ 2330.120923] LustreError: 126795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2337.249575] LustreError: 126781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.264824] LustreError: 126781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2338.272926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2338.278853] Lustre: Skipped 5 previous similar messages [ 2342.367152] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2342.756251] Lustre: server umount lustre-MDT0001 complete [ 2351.774097] Lustre: server umount lustre-OST0000 complete [ 2360.744710] Lustre: server umount lustre-OST0001 complete [ 2365.889827] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2372.965607] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2396.771760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.056323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2397.083237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2397.135622] Lustre: lustre-MDT0000: new disk, initializing [ 2397.218863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.225893] Lustre: Skipped 5 previous similar messages [ 2397.245751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2400.658308] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2408.185640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.281928] Lustre: 135430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2408.335919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2408.339930] Lustre: Skipped 1 previous similar message [ 2408.396774] Lustre: lustre-MDT0001: new disk, initializing [ 2408.497839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2408.503039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2411.479611] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2417.983498] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2421.543900] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2426.275108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2426.516611] Lustre: lustre-OST0000: new disk, initializing [ 2426.520188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2426.524371] Lustre: 137396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2428.051849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2428.073517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2428.213233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2431.219490] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2438.808646] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2445.837786] Lustre: lustre-MDD0000: changelog on [ 2447.812966] Lustre: lustre-MDD0001: changelog on [ 3575.981921] Lustre: lustre-MDD0001: changelog off [ 3576.810301] Lustre: 138840:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3577.201314] Lustre: lustre-MDD0000: changelog off [ 3580.385164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.388260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.389991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.393330] Lustre: Skipped 1 previous similar message [ 3585.150415] Lustre: server umount lustre-OST0000 complete [ 3590.624813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.625644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.630236] Lustre: Skipped 1 previous similar message [ 3590.635137] Lustre: Skipped 2 previous similar messages [ 3595.744610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.747252] Lustre: Skipped 1 previous similar message [ 3600.863127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3600.985239] Lustre: server umount lustre-MDT0000 complete [ 3602.457504] LustreError: 135423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467687 with bad export cookie 9423233071707800811 [ 3602.458673] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.463451] LustreError: 135423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3602.605116] Lustre: server umount lustre-MDT0001 complete [ 3607.009649] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3608.303838] Key type lgssc unregistered [ 3608.441592] LNet: 146421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3608.446452] LNetError: 146421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3608.454423] LNet: Removed LNI 192.168.204.146@tcp [ 3608.843399] Key type .llcrypt unregistered [ 3608.845416] Key type ._llcrypt unregistered [ 3617.527088] Key type ._llcrypt registered [ 3617.528786] Key type .llcrypt registered [ 3617.580077] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3625.421173] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3625.895438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3625.905229] alg: No test for adler32 (adler32-zlib) [ 3626.794950] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 3626.916305] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3628.511165] Key type lgssc registered [ 3628.968305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3643.996320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3644.008636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3645.094016] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3645.104706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3645.140559] Lustre: lustre-MDT0000: new disk, initializing [ 3645.166552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.175862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3646.823867] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3651.757308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.806391] Lustre: 149594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3651.834474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3651.836770] Lustre: Skipped 1 previous similar message [ 3651.874654] Lustre: lustre-MDT0001: new disk, initializing [ 3651.916089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3651.929804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3651.933226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3653.455078] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3657.741149] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3659.957620] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3662.607352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3662.723182] Lustre: lustre-OST0000: new disk, initializing [ 3662.725473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3662.729431] Lustre: 151562:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3662.762619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.296333] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3666.178286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3666.184600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3666.225589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3669.508915] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3672.016412] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module obdecho/obdecho [ 3672.692957] LustreError: 152566:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3672.697423] LustreError: 152566:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3673.331409] LustreError: 152662:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3677.663951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3677.667100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.671583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3681.761628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.762243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3681.766283] Lustre: Skipped 1 previous similar message [ 3681.770426] Lustre: Skipped 2 previous similar messages [ 3682.929894] Lustre: server umount lustre-MDT0000 complete [ 3684.564654] LustreError: 149586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467770 with bad export cookie 13934290521344288618 [ 3684.568357] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3684.570435] LustreError: 149586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3684.691320] Lustre: server umount lustre-MDT0001 complete [ 3696.482040] Lustre: server umount lustre-OST0000 complete [ 3704.450062] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3708.455243] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3722.349844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.437980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3722.449760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3722.487686] Lustre: lustre-MDT0000: new disk, initializing [ 3722.518230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.526602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3724.061248] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3728.634949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.684083] Lustre: 156369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3728.711515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3728.713613] Lustre: Skipped 1 previous similar message [ 3728.754176] Lustre: lustre-MDT0001: new disk, initializing [ 3728.794545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3728.809594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3728.813818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3730.386624] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3734.578660] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3736.796378] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3739.450925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3739.561472] Lustre: lustre-OST0000: new disk, initializing [ 3739.563298] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3739.567012] Lustre: 158336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3739.604524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.848588] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3746.098349] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3746.816149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3746.822323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3746.874281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3747.783630] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3747.866916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3749.537895] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3749.621845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3751.905902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3751.906160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.908133] LustreError: Skipped 1 previous similar message [ 3751.912307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3751.915724] Lustre: Skipped 1 previous similar message [ 3756.447300] Lustre: server umount lustre-OST0000 complete [ 3759.583880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3759.586932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.590947] Lustre: Skipped 1 previous similar message [ 3759.593247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.264852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.267738] Lustre: Skipped 3 previous similar messages [ 3772.383247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3772.473305] Lustre: server umount lustre-MDT0000 complete [ 3774.043089] LustreError: 156360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467859 with bad export cookie 13934290521344289577 [ 3774.047809] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3774.048092] LustreError: 156360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3774.205437] Lustre: server umount lustre-MDT0001 complete [ 3775.677510] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 13:04:20 (1786467860) [ 3776.312937] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3777.036922] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 13:04:21 (1786467861) [ 3780.079244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.328663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.015657] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3785.644654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.697305] LustreError: 160747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3785.706036] LustreError: 160747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3787.536405] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3790.325259] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3792.498580] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3795.161331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3797.579763] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3800.486799] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3809.763750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.766088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3809.770135] Lustre: Skipped 3 previous similar messages [ 3809.772314] Lustre: Skipped 3 previous similar messages [ 3814.636182] Lustre: server umount lustre-MDT0000 complete [ 3814.880884] LustreError: 161468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3814.888107] LustreError: 161468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3816.086300] LustreError: 160728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467901 with bad export cookie 13934290521344290445 [ 3816.089670] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.091991] LustreError: 160728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3816.210769] LustreError: 160742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.000097] LustreError: 160742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.000345] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.261208] Lustre: server umount lustre-MDT0001 complete [ 3824.662188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3829.494277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.586283] LustreError: 164584:0:(llog.c:1655:llog_backup()) MGC192.168.204.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3829.589669] Lustre: 164584:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3845.600434] LustreError: 164589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.607150] LustreError: 164589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3855.840058] LustreError: 164591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.845753] LustreError: 164591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3872.530373] LustreError: 164612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.537097] LustreError: 164612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 3890.272729] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3890.275314] Lustre: 164584:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3907.040329] LustreError: 164589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3907.048290] LustreError: 164589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 3931.111230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3931.114586] Lustre: Skipped 2 previous similar messages [ 3932.663625] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3936.104671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.227032] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3937.183113] LustreError: 165327:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3937.186093] LustreError: 165327:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000001f569895 x1873247167674752/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786468022 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3937.194986] LustreError: 165327:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3937.247825] LustreError: 147819:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f5d5f703 x1873247167678976/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3939.099916] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3942.055090] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3942.374721] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3943.773348] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 3946.215948] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3948.946709] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3957.729991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.730515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3957.731253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3957.731257] Lustre: Skipped 6 previous similar messages [ 3957.735818] Lustre: Skipped 1 previous similar message [ 3959.883393] Lustre: server umount lustre-MDT0000 complete [ 3961.393388] LustreError: 160729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468046 with bad export cookie 13934290521344296885 [ 3961.396635] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3961.398763] LustreError: 160729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3961.515522] Lustre: server umount lustre-MDT0001 complete [ 3973.285195] Lustre: server umount lustre-OST0000 complete [ 3982.085200] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3986.134734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4000.357971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4000.459707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4000.479310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4000.520035] Lustre: lustre-MDT0000: new disk, initializing [ 4000.558886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.562380] Lustre: Skipped 1 previous similar message [ 4000.571039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4002.315328] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4007.119869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4007.165462] Lustre: 169656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4007.271524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4008.795764] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4012.913525] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4015.095324] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4017.868286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4017.986071] Lustre: 171625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4020.315884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4023.814720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4023.859819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4024.593768] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4026.266179] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4026.361539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4028.075893] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4028.168890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4034.019037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4034.019374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4034.031439] Lustre: Skipped 1 previous similar message [ 4034.035939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4034.039232] Lustre: Skipped 3 previous similar messages [ 4043.231115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4043.293191] Lustre: server umount lustre-OST0000 complete [ 4048.352227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4048.357831] LustreError: Skipped 1 previous similar message [ 4059.103269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4059.175137] Lustre: server umount lustre-MDT0000 complete [ 4059.617644] LustreError: 170460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.624630] LustreError: 170460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 4060.713462] LustreError: 169648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468146 with bad export cookie 13934290521344313692 [ 4060.717360] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4060.718058] LustreError: 169648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4060.844446] Lustre: server umount lustre-MDT0001 complete [ 4064.306687] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 13:09:08 (1786468148) [ 4066.781973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.526352] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4071.395369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.976346] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4075.631362] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4077.488418] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4079.964822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4082.053504] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4084.549216] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4099.311993] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 13:09:44 (1786468184) [ 4101.600130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4101.604279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.610499] Lustre: Skipped 3 previous similar messages [ 4101.612659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4101.615809] Lustre: Skipped 7 previous similar messages [ 4106.575799] Lustre: server umount lustre-OST0000 complete [ 4114.137670] Lustre: server umount lustre-MDT0000 complete [ 4115.357572] LustreError: 174020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468200 with bad export cookie 13934290521344314546 [ 4115.360927] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4115.363316] LustreError: 174020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4115.479397] Lustre: server umount lustre-MDT0001 complete [ 4119.314722] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4120.365363] Key type lgssc unregistered [ 4120.500324] LNet: 177785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4120.504274] LNetError: 177785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4120.514359] LNet: Removed LNI 192.168.204.146@tcp [ 4120.843106] Key type .llcrypt unregistered [ 4120.844774] Key type ._llcrypt unregistered [ 4128.769806] Key type ._llcrypt registered [ 4128.771214] Key type .llcrypt registered [ 4128.826436] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4129.217252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4129.272745] alg: No test for adler32 (adler32-zlib) [ 4130.136334] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4130.247352] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4131.839241] Key type lgssc registered [ 4132.252878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4135.104529] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4135.113298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.559424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4155.679338] LustreError: 179004:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 4160.799453] LustreError: 179004:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4160.804151] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4160.806706] LustreError: MGC192.168.204.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4160.810721] LustreError: 179004:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4160.814578] LustreError: 179004:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4160.817694] LustreError: 179004:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4160.842160] Lustre: server umount lustre-OST0000 complete [ 4160.843809] LustreError: 179004:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4163.849762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4165.004479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4166.424687] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4169.300235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4169.439314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.965248] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4173.787704] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4175.968304] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4179.027827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.135806] LustreError: 181316:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4179.141489] Lustre: 181316:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4179.208200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.501295] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4184.420712] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4185.275450] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.282491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.285731] Lustre: Skipped 1 previous similar message [ 4190.177022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4190.180958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.187976] Lustre: Skipped 1 previous similar message [ 4190.190549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4191.448167] Lustre: server umount lustre-MDT0000 complete [ 4192.935683] LustreError: 179484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468278 with bad export cookie 1010031065915758004 [ 4192.940608] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4192.942070] LustreError: 179484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4193.080175] Lustre: server umount lustre-MDT0001 complete [ 4204.959039] Lustre: server umount lustre-OST0000 complete [ 4208.663822] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4213.143685] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4225.740544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.829050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4225.841979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4225.879974] Lustre: lustre-MDT0000: new disk, initializing [ 4225.904901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.911732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4227.437984] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4231.858857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.905986] Lustre: 184835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4231.934831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4231.936867] Lustre: Skipped 1 previous similar message [ 4231.973185] Lustre: lustre-MDT0001: new disk, initializing [ 4232.017535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4232.034301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4232.038873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4233.642453] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4237.707433] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4240.114989] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4243.023747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4243.145644] Lustre: lustre-OST0000: new disk, initializing [ 4243.147851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4243.151322] Lustre: 186801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4243.194382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.565442] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4248.067613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4248.073803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4248.105402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4249.962238] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4251.665942] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4251.757193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4253.247921] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4253.322244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4258.273896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4258.275162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.276241] LustreError: Skipped 1 previous similar message [ 4258.280556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4258.284027] Lustre: Skipped 3 previous similar messages [ 4260.119068] Lustre: server umount lustre-OST0000 complete [ 4262.880273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4262.883832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4262.888531] Lustre: Skipped 1 previous similar message [ 4262.890914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4267.603903] Lustre: server umount lustre-MDT0000 complete [ 4268.511952] LustreError: 186198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.517724] LustreError: 186198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4268.914381] LustreError: 184826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468354 with bad export cookie 1010031065915758697 [ 4268.917470] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4268.919267] LustreError: 184826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4269.050623] Lustre: server umount lustre-MDT0001 complete [ 4272.474113] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 13:12:37 (1786468357) [ 4279.124846] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4280.251251] Key type lgssc unregistered [ 4280.389536] LNet: 189743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4280.392923] LNetError: 189743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4280.400516] LNet: Removed LNI 192.168.204.146@tcp [ 4280.765510] Key type .llcrypt unregistered [ 4280.767123] Key type ._llcrypt unregistered [ 4288.376804] Key type ._llcrypt registered [ 4288.378584] Key type .llcrypt registered [ 4288.427460] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4288.782123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4288.831339] alg: No test for adler32 (adler32-zlib) [ 4289.689631] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4289.782382] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4291.359205] Key type lgssc registered [ 4291.770338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4294.558874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4294.565589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4295.699709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.998826] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4299.853677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.984473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4301.548441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4304.005890] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4305.873141] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4308.545674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4308.623148] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4308.624891] LustreError: 192798:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4308.627842] Lustre: 192798:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4308.670891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4310.652372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4313.062044] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4315.104811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.107027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4315.107696] Lustre: Skipped 2 previous similar messages [ 4315.109078] Lustre: Skipped 1 previous similar message [ 4319.834907] Lustre: server umount lustre-MDT0000 complete [ 4320.224641] LustreError: 190937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.230689] LustreError: 190937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4321.104135] LustreError: 190917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468406 with bad export cookie 9834308622261826811 [ 4321.106875] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4321.107677] LustreError: 190917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4321.222528] Lustre: server umount lustre-MDT0001 complete [ 4332.826373] Lustre: server umount lustre-OST0000 complete [ 4336.864299] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4340.574176] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4352.929936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.036645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4353.050523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4353.090328] Lustre: lustre-MDT0000: new disk, initializing [ 4353.133326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.144579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4354.745138] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4359.254541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.302956] Lustre: 196318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4359.330967] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4359.332712] Lustre: Skipped 1 previous similar message [ 4359.334187] LustreError: 197099:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4359.337419] LustreError: 196329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.338719] LustreError: 197099:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4359.348857] Lustre: 197099:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4359.359787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4359.362566] Lustre: Skipped 1 previous similar message [ 4359.400610] Lustre: lustre-MDT0001: new disk, initializing [ 4359.443992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4359.462081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4359.467088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4360.889355] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4364.693283] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4366.595664] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4368.873023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4368.997018] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4368.998697] Lustre: Skipped 1 previous similar message [ 4369.000339] LustreError: 198263:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4369.004237] LustreError: 198263:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4369.006900] Lustre: 198263:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4369.009938] Lustre: 198263:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4369.019456] Lustre: lustre-OST0000: new disk, initializing [ 4369.021724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4369.024839] Lustre: 198287:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4369.056559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.128656] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4374.010418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4374.014104] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4374.046303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4374.879667] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4376.410614] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4376.486990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4378.048418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4378.119135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4379.106021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.106274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.108285] LustreError: Skipped 1 previous similar message [ 4379.111860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.115560] Lustre: Skipped 1 previous similar message [ 4380.878238] Lustre: server umount lustre-OST0000 complete [ 4384.225406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.226275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4384.230211] Lustre: Skipped 2 previous similar messages [ 4388.251089] Lustre: server umount lustre-MDT0000 complete [ 4389.344641] LustreError: 196329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.349470] LustreError: 196329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4389.529471] LustreError: 197115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468475 with bad export cookie 9834308622261827511 [ 4389.532186] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.534117] LustreError: 197115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4389.638372] Lustre: server umount lustre-MDT0001 complete [ 4392.518145] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 13:14:37 (1786468477) [ 4397.644452] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4401.104720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.227190] LustreError: 201406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4401.239762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.540429] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4405.309621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.669175] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4407.542238] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4409.514752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.446663] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4414.118505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.182691] Lustre: lustre-OST0001: new disk, initializing [ 4414.184212] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4414.186541] Lustre: 203714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4416.063466] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4419.071949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4419.075307] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4419.106779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4421.306563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4427.876598] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4428.479864] Lustre: DEBUG MARKER: ADD OST3 [ 4430.033593] loop: module loaded [ 4430.051469] loop0: detected capacity change from 0 to 204800000 [ 4430.110437] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4430.113549] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4430.116601] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4432.234934] loop0: detected capacity change from 0 to 204800000 [ 4434.093748] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4434.159892] Lustre: lustre-OST0003: new disk, initializing [ 4434.161861] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4434.164794] Lustre: 205442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4434.203680] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4434.205714] Lustre: Skipped 3 previous similar messages [ 4435.107945] Lustre: DEBUG MARKER: STOP OST3 [ 4435.348752] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4435.386202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4435.390298] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4435.402138] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4435.868629] Lustre: Failing over lustre-OST0003 [ 4435.908755] Lustre: server umount lustre-OST0003 complete [ 4436.379281] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4437.407075] LustreError: 205682:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4440.490939] Lustre: DEBUG MARKER: START OST3 again [ 4440.551122] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.551163] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4440.551710] LustreError: 202894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.556729] Lustre: Skipped 1 previous similar message [ 4443.028600] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4443.146209] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4444.631311] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4444.848232] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4444.848319] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4445.452932] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4458.787900] Lustre: Failing over lustre-OST0003 [ 4458.819283] Lustre: server umount lustre-OST0003 complete [ 4460.512297] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4460.512332] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.515526] LustreError: Skipped 1 previous similar message [ 4460.515983] LustreError: 202891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.521814] Lustre: Skipped 1 previous similar message [ 4460.522922] LustreError: 202891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4472.287930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4472.291135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.296401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4472.299269] Lustre: Skipped 1 previous similar message [ 4477.214363] Lustre: server umount lustre-MDT0000 complete [ 4478.405084] LustreError: 201387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468563 with bad export cookie 9834308622261828393 [ 4478.410540] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.944138] LustreError: 201402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.945772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4478.948800] LustreError: 201402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4478.953488] Lustre: Skipped 5 previous similar messages [ 4484.658237] Lustre: server umount lustre-MDT0001 complete [ 4485.924153] Lustre: server umount lustre-OST0000 complete [ 4487.135680] Lustre: server umount lustre-OST0001 complete [ 4489.003889] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4491.417495] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4502.785523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.863261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4502.874060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4502.911544] Lustre: lustre-MDT0000: new disk, initializing [ 4502.936406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4502.939413] Lustre: Skipped 1 previous similar message [ 4502.945603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4504.436043] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4508.430875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.475336] Lustre: 210854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4508.568889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4509.776321] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4513.335744] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4515.055639] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4517.314565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4517.413020] Lustre: 212823:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4518.710450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4518.743640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4519.269851] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4522.952590] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4524.328874] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4524.402329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4525.707371] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4525.783055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4527.010561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.011088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.013815] Lustre: Skipped 6 previous similar messages [ 4527.016021] Lustre: Skipped 2 previous similar messages [ 4528.459282] Lustre: server umount lustre-OST0000 complete [ 4535.831133] Lustre: server umount lustre-MDT0000 complete [ 4537.042194] LustreError: 211648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468622 with bad export cookie 9834308622261830647 [ 4537.044651] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4537.046923] LustreError: 211648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4537.168355] Lustre: server umount lustre-MDT0001 complete [ 4538.327926] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:17:03 (1786468623) [ 4543.354887] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4546.340917] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4556.985527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.083412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4557.093146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4557.095374] Lustre: Skipped 3 previous similar messages [ 4557.132804] Lustre: lustre-MDT0000: new disk, initializing [ 4557.134515] Lustre: Skipped 2 previous similar messages [ 4557.155163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4557.159187] Lustre: Skipped 2 previous similar messages [ 4558.419492] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4562.513135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.558812] Lustre: 217422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4562.658378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4564.023675] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4567.869920] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4569.839621] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4572.073921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4572.174183] Lustre: 219389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4574.004734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4574.049154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4577.493226] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4578.824022] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4578.892297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4580.141057] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4580.209830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4584.417066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4584.418987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.419395] LustreError: Skipped 1 previous similar message [ 4584.424665] Lustre: Skipped 2 previous similar messages [ 4584.425240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4584.429362] Lustre: Skipped 3 previous similar messages [ 4595.167067] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4595.212200] Lustre: server umount lustre-OST0000 complete [ 4598.751790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4602.453871] Lustre: server umount lustre-MDT0000 complete [ 4603.591222] LustreError: 219400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468689 with bad export cookie 9834308622261831529 [ 4603.594285] LustreError: 219400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4603.594316] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4603.708589] Lustre: server umount lustre-MDT0001 complete [ 4606.364364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.504617] LustreError: 221603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.510148] LustreError: 221603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4607.793523] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4610.392160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.766405] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4613.992699] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4615.717835] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4617.739817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4619.515697] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4621.799328] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4640.735895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4640.739429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4640.741753] Lustre: Skipped 7 previous similar messages [ 4644.740515] Lustre: server umount lustre-OST0000 complete [ 4660.191121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4660.248984] Lustre: server umount lustre-MDT0000 complete [ 4661.422180] LustreError: 221585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468746 with bad export cookie 9834308622261832411 [ 4661.424221] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4661.425811] LustreError: 221585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4661.548454] Lustre: server umount lustre-MDT0001 complete [ 4754.530721] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4755.547157] Key type lgssc unregistered [ 4755.666333] LNet: 225291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4755.670048] LNetError: 225291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4755.677257] LNet: Removed LNI 192.168.204.146@tcp [ 4755.984109] Key type .llcrypt unregistered [ 4755.985190] Key type ._llcrypt unregistered [ 4763.949648] Key type ._llcrypt registered [ 4763.950678] Key type .llcrypt registered [ 4763.989139] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4770.296499] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4770.609476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4770.649486] alg: No test for adler32 (adler32-zlib) [ 4771.504986] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4771.592527] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4773.175190] Key type lgssc registered [ 4773.571083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4785.033939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4785.039316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.118133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4786.131161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4786.169175] Lustre: lustre-MDT0000: new disk, initializing [ 4786.190925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.198145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4787.553356] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4791.673365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.713760] Lustre: 228616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4791.737419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4791.739973] Lustre: Skipped 1 previous similar message [ 4791.776462] Lustre: lustre-MDT0001: new disk, initializing [ 4791.810082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4791.822951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4791.827177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4793.142173] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4796.752551] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4798.591285] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4800.938615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.043043] Lustre: lustre-OST0000: new disk, initializing [ 4801.044548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4801.046671] Lustre: 230581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4801.075410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4802.985659] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4806.701685] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4807.159281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4807.162458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4807.193667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4808.091953] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4808.161446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4809.457311] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4809.525917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4812.258124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4812.259191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.261069] LustreError: Skipped 1 previous similar message [ 4812.267454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4812.270484] Lustre: Skipped 1 previous similar message [ 4816.275399] Lustre: server umount lustre-OST0000 complete [ 4822.497312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4822.497817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4822.498580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4822.507028] Lustre: Skipped 2 previous similar messages [ 4823.639509] Lustre: server umount lustre-MDT0000 complete [ 4824.868248] LustreError: 229410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468910 with bad export cookie 2561014950560178776 [ 4824.871554] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4824.874814] LustreError: 229410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4824.988743] Lustre: server umount lustre-MDT0001 complete [ 4831.016802] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4834.342584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.490248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.901565] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4838.832712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.950135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4840.340710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4841.324984] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4843.583583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.752923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4848.793520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4848.867071] Lustre: lustre-OST0001: new disk, initializing [ 4848.869474] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4848.872177] Lustre: 235815:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4848.908975] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4848.911508] Lustre: Skipped 1 previous similar message [ 4850.968942] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4853.758842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4853.761408] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4853.795994] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4856.427398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4859.032292] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4862.809764] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:22:27 (1786468947) [ 4863.971602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.975072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4863.975929] Lustre: Skipped 3 previous similar messages [ 4863.977897] Lustre: Skipped 3 previous similar messages [ 4869.088541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4869.091125] Lustre: Skipped 3 previous similar messages [ 4869.658223] Lustre: server umount lustre-MDT0000 complete [ 4870.881205] LustreError: 236582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468956 with bad export cookie 2561014950560179644 [ 4870.883263] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4870.885671] LustreError: 236582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4870.993569] Lustre: server umount lustre-MDT0001 complete [ 4882.438755] Lustre: server umount lustre-OST0000 complete [ 4893.975552] Lustre: server umount lustre-OST0001 complete [ 4895.916958] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4898.972057] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4910.544742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4910.643989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4910.654818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4910.689905] Lustre: lustre-MDT0000: new disk, initializing [ 4910.712383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4910.719031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4911.985617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4916.035458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4916.080762] Lustre: 239752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4916.178965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4917.524860] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4921.143511] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4922.899737] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4925.117855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4925.209870] Lustre: lustre-OST0000: new disk, initializing [ 4925.210968] Lustre: Skipped 1 previous similar message [ 4925.212339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4925.214186] Lustre: Skipped 2 previous similar messages [ 4925.216217] Lustre: 241722:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4926.454619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4926.457089] Lustre: Skipped 1 previous similar message [ 4926.460684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4926.487146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4927.065886] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4930.596503] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4931.899941] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4931.970759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4933.263689] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4933.331396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4936.673363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4936.673967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.675641] LustreError: Skipped 1 previous similar message [ 4936.680810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4936.682628] Lustre: Skipped 1 previous similar message [ 4940.061323] Lustre: server umount lustre-OST0000 complete [ 4941.792520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.792593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4941.795926] Lustre: Skipped 2 previous similar messages [ 4946.912169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.915230] Lustre: Skipped 3 previous similar messages [ 4947.419995] Lustre: server umount lustre-MDT0000 complete [ 4948.769189] LustreError: 240546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469034 with bad export cookie 2561014950560181737 [ 4948.772742] LustreError: 240546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4948.773110] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.892615] Lustre: server umount lustre-MDT0001 complete [ 4951.836335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.995984] LustreError: 243932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.003336] LustreError: 243932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4952.023716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.025643] Lustre: Skipped 2 previous similar messages [ 4953.393814] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4956.244120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.714572] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4960.104210] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4961.915923] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4964.065031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4966.045844] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4968.386902] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5114.848103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.851450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.857208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5127.135215] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5127.232685] Lustre: server umount lustre-OST0000 complete [ 5130.720384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5130.727341] LustreError: Skipped 1 previous similar message [ 5131.233239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.241124] Lustre: Skipped 2 previous similar messages [ 5135.547696] Lustre: server umount lustre-MDT0000 complete [ 5136.354052] LustreError: 243932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.361717] LustreError: 243932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5137.490553] LustreError: 243914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469222 with bad export cookie 2561014950560182619 [ 5137.491375] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5137.495500] LustreError: 243914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5137.683760] Lustre: server umount lustre-MDT0001 complete [ 5234.994744] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5236.606769] Key type lgssc unregistered [ 5236.793724] LNet: 247543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5236.797654] LNetError: 247543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5236.810487] LNet: Removed LNI 192.168.204.146@tcp [ 5237.292220] Key type .llcrypt unregistered [ 5237.296584] Key type ._llcrypt unregistered [ 5251.054584] Key type ._llcrypt registered [ 5251.056086] Key type .llcrypt registered [ 5251.129398] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5260.658538] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5261.231736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5261.245835] alg: No test for adler32 (adler32-zlib) [ 5262.181278] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 5262.322282] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5263.935303] Key type lgssc registered [ 5264.561986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5288.776768] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5294.051540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5294.083398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.210637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5295.231816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5295.274856] Lustre: lustre-MDT0000: new disk, initializing [ 5295.324703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.337733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5297.520609] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5303.186482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.243976] Lustre: 251586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5303.276263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5303.280420] Lustre: Skipped 1 previous similar message [ 5303.323113] Lustre: lustre-MDT0001: new disk, initializing [ 5303.367380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5303.385108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5303.389538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5305.652437] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5308.645865] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5311.856739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5312.016631] Lustre: lustre-OST0000: new disk, initializing [ 5312.019225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5312.025990] Lustre: 253218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5312.096037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5315.444534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5318.679988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5318.685588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5318.754122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5321.930806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5322.022682] Lustre: lustre-OST0001: new disk, initializing [ 5322.025757] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5322.030970] Lustre: 254091:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5322.083881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5325.112684] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5326.909142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5326.913737] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5326.973977] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5333.213437] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5337.051679] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5344.224468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5344.229654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.238987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5347.809071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5347.812098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.819083] Lustre: Skipped 1 previous similar message [ 5349.460052] Lustre: server umount lustre-MDT0000 complete [ 5351.431434] LustreError: 252382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469436 with bad export cookie 16245219640575336192 [ 5351.437757] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5351.438371] LustreError: 252382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5351.620128] Lustre: server umount lustre-MDT0001 complete [ 5363.873444] Lustre: server umount lustre-OST0000 complete [ 5376.509719] Lustre: server umount lustre-OST0001 complete [ 5383.401526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5388.295292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5393.133493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5398.920718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5409.195975] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5413.919919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.934247] Lustre: lustre-MDT0000: reset Object Index mappings [ 5414.189345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.415725] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5419.495502] LustreError: 259094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5419.507532] LustreError: 259094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5420.940927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.956939] Lustre: lustre-MDT0001: reset Object Index mappings [ 5423.378807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5424.837788] Lustre: 260233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5428.198195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.389637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.392780] Lustre: Skipped 1 previous similar message [ 5431.215461] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5431.463405] LustreError: 260591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.306847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5438.435178] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5439.545630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5439.556160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5443.524795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5445.829211] Lustre: 262104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5449.700347] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.703243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.706517] Lustre: Skipped 4 previous similar messages [ 5449.714390] Lustre: Skipped 6 previous similar messages [ 5454.074712] Lustre: server umount lustre-MDT0000 complete [ 5454.820349] LustreError: 259093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.830062] LustreError: 259093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5455.948483] LustreError: 262106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469541 with bad export cookie 16245219640575342968 [ 5455.951705] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.956177] LustreError: 262106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5456.122181] Lustre: server umount lustre-MDT0001 complete [ 5468.203572] Lustre: server umount lustre-OST0000 complete [ 5479.653262] Lustre: server umount lustre-OST0001 complete [ 5489.368446] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:32:53 (1786469573) [ 5497.654399] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5503.785904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5522.242558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.370878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5522.388642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5522.432832] Lustre: lustre-MDT0000: new disk, initializing [ 5522.484171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5522.487121] Lustre: Skipped 1 previous similar message [ 5522.496572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5524.569155] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5530.381612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.433520] Lustre: 266353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5530.473631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5530.476276] Lustre: Skipped 1 previous similar message [ 5530.518189] Lustre: lustre-MDT0001: new disk, initializing [ 5530.584916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5530.589366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5532.396108] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5537.177540] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5539.744543] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5542.901428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5543.051195] Lustre: 268322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5544.665803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5544.720892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5545.944654] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5550.903387] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5552.679427] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5552.801300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5554.585052] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5554.683454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5560.289433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5560.290772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.291913] LustreError: Skipped 1 previous similar message [ 5560.298642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5560.301104] Lustre: Skipped 1 previous similar message [ 5561.638397] Lustre: server umount lustre-OST0000 complete [ 5565.410205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.412315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5565.416079] Lustre: Skipped 2 previous similar messages [ 5565.421356] Lustre: Skipped 1 previous similar message [ 5569.714691] Lustre: server umount lustre-MDT0000 complete [ 5570.529148] LustreError: 266359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.538472] LustreError: 266359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5571.382652] LustreError: 266345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469656 with bad export cookie 16245219640575345054 [ 5571.387083] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5571.388743] LustreError: 266345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5571.522804] Lustre: server umount lustre-MDT0001 complete [ 5579.220106] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5583.564033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.755367] LustreError: 271252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.775584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.778571] Lustre: Skipped 2 previous similar messages [ 5585.411238] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5588.933623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.807114] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5592.053479] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5594.908448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5597.787843] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5601.660880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.739205] Lustre: lustre-OST0001: new disk, initializing [ 5601.741075] Lustre: Skipped 1 previous similar message [ 5601.743589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5601.746221] Lustre: Skipped 1 previous similar message [ 5601.749565] Lustre: 273558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5603.186838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5603.197928] Lustre: Skipped 1 previous similar message [ 5603.200981] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5603.255331] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5604.616994] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5610.033591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5611.824370] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5614.352878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5614.362515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5619.682039] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.686588] Lustre: Skipped 2 previous similar messages [ 5619.687527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5619.688751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5619.690923] Lustre: Skipped 3 previous similar messages [ 5623.180814] Lustre: server umount lustre-MDT0000 complete [ 5624.802683] LustreError: 271252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5624.812096] LustreError: 271252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5624.941642] LustreError: 271233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469710 with bad export cookie 16245219640575345929 [ 5624.946926] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5624.948434] LustreError: 271233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5625.106291] Lustre: server umount lustre-MDT0001 complete [ 5637.149624] Lustre: server umount lustre-OST0000 complete [ 5649.249742] Lustre: server umount lustre-OST0001 complete [ 5656.233031] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5660.717060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.911735] LustreError: 276606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.942701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5660.945508] Lustre: Skipped 3 previous similar messages [ 5662.770969] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5666.806915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.968301] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5670.302199] Lustre: 277746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5673.306975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5675.977359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5680.015600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5682.937908] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5686.256764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5686.272881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5691.860091] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5693.352664] Lustre: 279621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5695.599943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5695.625939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5699.584821] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:36:24 (1786469784) [ 5711.841243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.844600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.847460] Lustre: Skipped 2 previous similar messages [ 5711.852965] Lustre: Skipped 3 previous similar messages [ 5713.585705] Lustre: server umount lustre-MDT0000 complete [ 5715.133514] LustreError: 280329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469800 with bad export cookie 16245219640575352236 [ 5715.137420] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.138948] LustreError: 280329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5716.962558] LustreError: 276602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5716.969517] LustreError: 276602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5717.315064] Lustre: server umount lustre-MDT0001 complete [ 5725.126366] Lustre: server umount lustre-OST0000 complete [ 5732.848985] Lustre: server umount lustre-OST0001 complete [ 5735.722639] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5740.041358] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5759.468190] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5763.887882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5764.001223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5764.021401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5764.067967] Lustre: lustre-MDT0000: new disk, initializing [ 5764.105505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5765.909663] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5771.283479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.340675] Lustre: 283862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5771.467635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5773.367958] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5778.848841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5778.971638] Lustre: 285495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5780.239898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5781.439197] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5786.482489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5787.850482] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5789.134338] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5794.040101] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5800.338892] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:38:04 (1786469884) [ 5802.463826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5802.466892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.471637] Lustre: Skipped 3 previous similar messages [ 5802.474587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5802.477051] Lustre: Skipped 2 previous similar messages [ 5807.337885] Lustre: server umount lustre-MDT0000 complete [ 5808.608430] LustreError: 283874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5808.615357] LustreError: 283874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5808.845579] LustreError: 283854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469894 with bad export cookie 16245219640575360125 [ 5808.846490] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5808.850647] LustreError: 283854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5808.995557] Lustre: server umount lustre-MDT0001 complete [ 5820.940889] Lustre: server umount lustre-OST0000 complete [ 5832.885778] Lustre: server umount lustre-OST0001 complete [ 5836.780744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.054703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5837.058853] Lustre: Skipped 7 previous similar messages [ 5839.008567] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5842.869689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.923449] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5848.350338] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5850.844857] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5854.097959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.067872] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5860.299522] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5876.230688] loop1: detected capacity change from 0 to 204800000 [ 5876.320603] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5876.324586] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5876.329148] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5878.210132] loop1: detected capacity change from 0 to 204800000 [ 5880.521029] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.557704] LustreError: 288597:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9614: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5880.564391] LustreError: 288597:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9614 [ 5880.568040] LustreError: 288597:0:(mgs_handler.c:517:mgs_target_reg()) fs9614-MDT0001: Error checking target index: rc = -13 [ 5880.572526] LustreError: fs9614-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5880.576318] LustreError: 292342:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5880.579598] LustreError: 292342:0:(tgt_mount.c:2129:server_put_super()) no obd fs9614-MDT0001 [ 5880.582530] LustreError: 292342:0:(tgt_mount.c:116:server_deregister_mount()) fs9614-MDT0001 not registered [ 5880.599038] Lustre: server umount fs9614-MDT0001 complete [ 5880.600338] LustreError: 292342:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5883.906540] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.941892] Lustre: Setting parameter fs9614-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9614-MDT0001 [ 5883.947847] Lustre: Skipped 2 previous similar messages [ 5883.971212] Lustre: srv-fs9614-MDT0001: No data found on store. Initialize space. [ 5883.973278] Lustre: Skipped 4 previous similar messages [ 5884.012156] Lustre: fs9614-MDT0001: new disk, initializing [ 5884.014255] Lustre: Skipped 3 previous similar messages [ 5884.059335] Lustre: srv-fs9614-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5884.064900] Lustre: 292807:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9614-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5884.071804] LustreError: 292807:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9614-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5885.595072] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5888.713279] Lustre: server umount fs9614-MDT0001 complete [ 5893.948181] loop1: detected capacity change from 0 to 204800000 [ 5896.392872] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.539108] Lustre: 294373:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9614-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5896.544777] LustreError: 294373:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9614-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5898.365486] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5900.498083] Lustre: server umount fs9614-MDT0001 complete [ 5906.721988] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 5906.727027] Lustre: Skipped 4 previous similar messages [ 5908.963051] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5908.970506] Lustre: Skipped 5 previous similar messages [ 5910.692247] Lustre: server umount lustre-MDT0000 complete [ 5914.304867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.373817] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5916.241048] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5919.492967] loop1: detected capacity change from 0 to 204800000 [ 5919.718046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5921.777187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.941048] Lustre: 296818:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9614-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5921.945280] LustreError: 296818:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9614-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5923.624373] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5925.691119] Lustre: server umount fs9614-MDT0001 complete [ 5934.560073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5940.195524] LustreError: 290438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.203140] LustreError: 290438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 49 previous similar messages [ 5948.383297] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5948.456551] Lustre: server umount lustre-OST0000 complete [ 5950.432553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5950.435876] LustreError: Skipped 1 previous similar message [ 5956.211315] Lustre: server umount lustre-MDT0000 complete [ 5957.868914] LustreError: 293351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470043 with bad export cookie 16245219640575362799 [ 5957.873102] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5957.874016] LustreError: 293351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5958.025755] Lustre: server umount lustre-MDT0001 complete [ 5962.655629] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5963.874162] Key type lgssc unregistered [ 5964.016549] LNet: 299162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5964.020626] LNetError: 299162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5964.029721] LNet: Removed LNI 192.168.204.146@tcp [ 5964.397170] Key type .llcrypt unregistered [ 5964.398901] Key type ._llcrypt unregistered [ 5973.178804] Key type ._llcrypt registered [ 5973.180027] Key type .llcrypt registered [ 5973.226450] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5973.606340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5973.647054] alg: No test for adler32 (adler32-zlib) [ 5974.521801] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 5974.625962] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5976.215178] Key type lgssc registered [ 5976.647909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5979.600430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5979.607279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5980.780141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5982.467896] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5985.848554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.038273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5987.634913] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5990.535535] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5992.691724] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5995.352721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5995.517306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.578828] LustreError: 302408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5996.586122] LustreError: 302408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5997.861622] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6000.876447] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6001.635410] LustreError: 302246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.729102] LustreError: 302245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.736766] LustreError: 302245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6010.850543] LustreError: 302246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.856218] LustreError: 302246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6013.898762] loop1: detected capacity change from 0 to 204800000 [ 6013.970374] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6013.975923] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6013.981913] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6015.972721] LustreError: 302246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6015.977692] loop1: detected capacity change from 0 to 204800000 [ 6015.978849] LustreError: 302246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6018.181907] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.240452] Lustre: fs22732-OST0000: new disk, initializing [ 6018.243236] Lustre: srv-fs22732-OST0000: No data found on store. Initialize space. [ 6018.246992] Lustre: 304150:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6018.273151] Lustre: fs22732-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.505040] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6024.423338] Lustre: server umount fs22732-OST0000 complete [ 6026.210950] LustreError: 302918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.219200] LustreError: 302918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6029.637049] loop1: detected capacity change from 0 to 204800000 [ 6032.093456] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6032.186690] Lustre: fs22732-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.602686] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6038.058271] loop2: detected capacity change from 0 to 204800000 [ 6038.140613] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6038.144802] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6038.150385] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6040.319772] loop2: detected capacity change from 0 to 204800000 [ 6042.855772] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6042.897372] LustreError: fs22732-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6042.901473] LustreError: fs22732-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6042.905449] LustreError: 307186:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6042.909447] LustreError: 307186:0:(tgt_mount.c:2129:server_put_super()) no obd fs22732-OST0001 [ 6042.912513] LustreError: 307186:0:(tgt_mount.c:116:server_deregister_mount()) fs22732-OST0001 not registered [ 6042.932559] Lustre: server umount fs22732-OST0001 complete [ 6042.935066] LustreError: 307186:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6043.412061] LustreError: 302246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.423798] LustreError: 302246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6047.517727] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6047.580924] Lustre: fs22732-OST0001: new disk, initializing [ 6047.583285] Lustre: srv-fs22732-OST0001: No data found on store. Initialize space. [ 6047.586436] Lustre: 307737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6047.619646] Lustre: fs22732-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6050.079451] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6053.985189] Lustre: server umount fs22732-OST0000 complete [ 6057.447330] Lustre: server umount fs22732-OST0001 complete [ 6062.107615] loop1: detected capacity change from 0 to 204800000 [ 6064.354858] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6064.455112] Lustre: fs22732-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6067.011766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6070.013252] loop2: detected capacity change from 0 to 204800000 [ 6072.539086] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6075.207344] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6077.410825] LustreError: 302918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6077.416259] LustreError: 302918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6078.771256] Lustre: server umount fs22732-OST0000 complete [ 6082.175340] Lustre: server umount fs22732-OST0001 complete [ 6087.652085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.652305] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6087.662875] LustreError: Skipped 1 previous similar message [ 6087.666796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6087.670340] Lustre: Skipped 1 previous similar message [ 6092.704482] Lustre: server umount lustre-OST0000 complete [ 6097.889327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.890851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6097.894793] Lustre: Skipped 2 previous similar messages [ 6097.899449] Lustre: Skipped 1 previous similar message [ 6100.451207] Lustre: server umount lustre-MDT0000 complete [ 6101.925179] LustreError: 300398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470187 with bad export cookie 7616738962302690033 [ 6101.927919] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6101.929064] LustreError: 300398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6102.065462] Lustre: server umount lustre-MDT0001 complete [ 6106.585836] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6107.811593] Key type lgssc unregistered [ 6107.953064] LNet: 313649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6107.956805] LNetError: 313649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.965591] LNet: Removed LNI 192.168.204.146@tcp [ 6108.334446] Key type .llcrypt unregistered [ 6108.336926] Key type ._llcrypt unregistered [ 6120.641598] Key type ._llcrypt registered [ 6120.642712] Key type .llcrypt registered [ 6120.688647] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6128.775146] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6129.335092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6129.356087] alg: No test for adler32 (adler32-zlib) [ 6130.282484] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6130.440727] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6132.047132] Key type lgssc registered [ 6132.641607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6149.814117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6149.827069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.960345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6150.984903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6151.027107] Lustre: lustre-MDT0000: new disk, initializing [ 6151.064791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.075152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6152.827915] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6158.071352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.129525] Lustre: 317480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6158.161658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6158.164207] Lustre: Skipped 1 previous similar message [ 6158.201976] Lustre: lustre-MDT0001: new disk, initializing [ 6158.242519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6158.259556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6158.266208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6160.085888] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6164.750326] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6167.233300] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6170.168687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6170.288966] Lustre: lustre-OST0000: new disk, initializing [ 6170.291326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6170.295165] Lustre: 319448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6170.334837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6172.818141] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6177.170256] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6180.037124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6180.041359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6180.082696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6185.463368] Lustre: Mounted lustre-client - version 2.17.56_50_g7b00149 [ 6185.899776] Lustre: Unmounted lustre-client [ 6190.562044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6190.563276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.567349] LustreError: Skipped 1 previous similar message [ 6190.575384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6190.581245] Lustre: Skipped 1 previous similar message [ 6195.683400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6195.686514] Lustre: Skipped 1 previous similar message [ 6195.804627] Lustre: server umount lustre-OST0000 complete [ 6199.264022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6199.266971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.272214] Lustre: Skipped 1 previous similar message [ 6199.274640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6200.802748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6203.754437] Lustre: server umount lustre-MDT0000 complete [ 6205.312508] LustreError: 317472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470290 with bad export cookie 4043670889295261419 [ 6205.315191] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6205.318272] LustreError: 317472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6205.453627] Lustre: server umount lustre-MDT0001 complete [ 6209.887626] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6211.125643] Key type lgssc unregistered [ 6236.895937] LNet: 321608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6236.899702] LNetError: 321608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6309.605701] LNet: Removed LNI 192.168.204.146@tcp [ 6310.028168] Key type .llcrypt unregistered [ 6310.029457] Key type ._llcrypt unregistered [ 6317.886858] Key type ._llcrypt registered [ 6317.888182] Key type .llcrypt registered [ 6317.931143] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6325.156525] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6325.657403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6325.666410] alg: No test for adler32 (adler32-zlib) [ 6326.557435] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6326.669283] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6328.255149] Key type lgssc registered [ 6328.716736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6340.887096] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:47:05 (1786470425) [ 6343.710702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6343.720146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6344.807578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6344.818049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6344.853751] Lustre: lustre-MDT0000: new disk, initializing [ 6344.901159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.907911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6346.374285] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6351.680486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.743708] Lustre: 324801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6351.778457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6351.781088] Lustre: Skipped 1 previous similar message [ 6351.821086] Lustre: lustre-MDT0001: new disk, initializing [ 6351.878814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6351.903871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6351.910838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6353.760106] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6358.300379] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6360.548487] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6363.219477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.329848] Lustre: lustre-OST0000: new disk, initializing [ 6363.331634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6363.334667] Lustre: 326770:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6363.367333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.633963] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6366.077921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6366.081724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6366.123230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6369.885314] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6376.418909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.419364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.420157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6376.424768] Lustre: Skipped 1 previous similar message [ 6376.436696] LustreError: Skipped 1 previous similar message [ 6381.538284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6381.540588] Lustre: Skipped 2 previous similar messages [ 6382.432715] Lustre: server umount lustre-OST0000 complete [ 6386.657718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.658925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6386.664321] Lustre: Skipped 1 previous similar message [ 6389.979923] Lustre: server umount lustre-MDT0000 complete [ 6391.367202] LustreError: 326768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470476 with bad export cookie 12014980490417402857 [ 6391.370519] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6391.371364] LustreError: 326768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6391.493909] Lustre: server umount lustre-MDT0001 complete [ 6395.704604] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6396.808657] Key type lgssc unregistered [ 6396.935610] LNet: 328678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6396.938857] LNetError: 328678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6396.947621] LNet: Removed LNI 192.168.204.146@tcp [ 6397.286572] Key type .llcrypt unregistered [ 6397.288270] Key type ._llcrypt unregistered [ 6406.618687] Key type ._llcrypt registered [ 6406.620224] Key type .llcrypt registered [ 6406.662980] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6407.026584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6407.077364] alg: No test for adler32 (adler32-zlib) [ 6407.973757] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6408.090192] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6409.687227] Key type lgssc registered [ 6410.109657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6413.149896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6413.160082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6414.326392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.917372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6419.335814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.493985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6421.175463] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6424.200698] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6426.403940] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6428.966894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6429.082985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.325892] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6434.236476] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6448.218371] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6448.844936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6450.972511] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 6454.753532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.755220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.758735] Lustre: Skipped 2 previous similar messages [ 6454.766484] Lustre: Skipped 2 previous similar messages [ 6456.083639] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 6456.537292] Lustre: server umount lustre-MDT0000 complete [ 6458.056364] LustreError: 331770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470543 with bad export cookie 13821072611727892507 [ 6458.059339] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6458.061689] LustreError: 331770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6459.872762] LustreError: 332427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.873266] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.878465] LustreError: 332427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6459.887108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6461.203576] LustreError: 329922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.904129] LustreError: 332427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.904597] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6463.914026] Lustre: Skipped 1 previous similar message [ 6464.263031] Lustre: server umount lustre-MDT0001 complete [ 6480.351129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6480.426288] Lustre: server umount lustre-OST0000 complete [ 6483.727690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.687272] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6486.994640] Lustre: 334127:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [ 6487.970505] Lustre: server umount MGS complete [ 6491.731712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.942884] LustreError: 334697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.977329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6493.664774] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6496.968406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.132608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6498.672562] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6502.017391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6504.450575] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6507.251507] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6507.842050] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6509.805138] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6517.556124] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6522.850633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6522.850888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.855165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6522.859018] Lustre: Skipped 1 previous similar message [ 6524.882452] Lustre: server umount lustre-OST0000 complete [ 6527.968967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6527.969194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.978102] Lustre: Skipped 1 previous similar message [ 6532.458801] Lustre: server umount lustre-MDT0000 complete [ 6533.088401] LustreError: 334694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.096408] LustreError: 334694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6533.944733] LustreError: 334680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470619 with bad export cookie 13821072611727893333 [ 6533.946536] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6533.952323] LustreError: 334680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6534.096145] Lustre: server umount lustre-MDT0001 complete [ 6538.469903] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6539.668435] Key type lgssc unregistered [ 6539.804581] LNet: 338135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6539.807963] LNetError: 338135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6539.817656] LNet: Removed LNI 192.168.204.146@tcp [ 6540.168343] Key type .llcrypt unregistered [ 6540.169924] Key type ._llcrypt unregistered [ 6547.085323] Key type ._llcrypt registered [ 6547.086465] Key type .llcrypt registered [ 6547.130063] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6554.272658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6554.658982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6554.690057] alg: No test for adler32 (adler32-zlib) [ 6555.577893] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6555.681824] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6557.271289] Key type lgssc registered [ 6557.716229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6570.671619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6570.679323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.765626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6571.780078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6571.815371] Lustre: lustre-MDT0000: new disk, initializing [ 6571.836846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6571.843556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6573.409218] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6578.142853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6578.198250] Lustre: 341266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6578.229523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6578.231367] Lustre: Skipped 1 previous similar message [ 6578.269209] Lustre: lustre-MDT0001: new disk, initializing [ 6578.314576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6578.328227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6578.332690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6580.039124] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6584.350123] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6586.671301] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6589.568402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6589.685967] Lustre: lustre-OST0000: new disk, initializing [ 6589.688928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6589.692789] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6589.697286] Lustre: 343241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6589.739788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6592.208625] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6595.055711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6595.063660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6595.088854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6596.676514] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6598.462493] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6598.556511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6600.217669] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6600.298348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6603.101552] Lustre: server umount lustre-OST0000 complete [ 6605.280489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.288520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6605.290966] Lustre: Skipped 1 previous similar message [ 6610.400242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.403084] Lustre: Skipped 1 previous similar message [ 6615.520207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6615.523325] Lustre: Skipped 1 previous similar message [ 6615.553526] Lustre: server umount lustre-MDT0000 complete [ 6617.095625] LustreError: 341260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470702 with bad export cookie 12289524051123837889 [ 6617.098506] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6617.102546] LustreError: 341260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6627.348421] Lustre: server umount lustre-MDT0001 complete [ 6630.742083] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:51:55 (1786470715) [ 6637.249141] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6641.076528] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6656.068858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.203754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6656.216897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6656.258959] Lustre: lustre-MDT0000: new disk, initializing [ 6656.299578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.309350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6658.079907] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6663.017486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.072723] Lustre: 347988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6663.104180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6663.106079] Lustre: Skipped 1 previous similar message [ 6663.143929] Lustre: lustre-MDT0001: new disk, initializing [ 6663.183503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.198723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6663.203153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6664.855277] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6669.197986] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6671.376711] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6674.227284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6674.357503] Lustre: lustre-OST0000: new disk, initializing [ 6674.360158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6674.364688] Lustre: 349956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6674.401893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6676.796858] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6681.100066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6681.104320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6681.153227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6681.365977] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6692.888402] Lustre: server umount lustre-OST0000 complete [ 6696.416989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.418179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6696.423279] Lustre: Skipped 2 previous similar messages [ 6696.427443] Lustre: Skipped 1 previous similar message [ 6703.623289] Lustre: server umount lustre-MDT0000 complete [ 6705.253319] LustreError: 348784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470790 with bad export cookie 12289524051123838757 [ 6705.255817] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6705.257596] LustreError: 348784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6714.911790] Lustre: server umount lustre-MDT0001 complete [ 6719.211964] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6720.410967] Key type lgssc unregistered [ 6720.544332] LNet: 352108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6720.548078] LNetError: 352108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6720.555486] LNet: Removed LNI 192.168.204.146@tcp [ 6720.923424] Key type .llcrypt unregistered [ 6720.924711] Key type ._llcrypt unregistered [ 6789.142213] Key type ._llcrypt registered [ 6789.144022] Key type .llcrypt registered [ 6789.205992] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6789.395974] Key type .llcrypt unregistered [ 6789.399062] Key type ._llcrypt unregistered [ 6829.386823] Key type ._llcrypt registered [ 6829.388589] Key type .llcrypt registered [ 6829.455911] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6829.625656] Key type .llcrypt unregistered [ 6829.627655] Key type ._llcrypt unregistered [ 6847.424907] Key type ._llcrypt registered [ 6847.427352] Key type .llcrypt registered [ 6847.500839] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6847.682468] Key type .llcrypt unregistered [ 6847.684046] Key type ._llcrypt unregistered [ 6865.439836] Key type ._llcrypt registered [ 6865.442210] Key type .llcrypt registered [ 6865.524283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6865.733654] Key type .llcrypt unregistered [ 6865.736087] Key type ._llcrypt unregistered [ 6913.864311] Key type ._llcrypt registered [ 6913.866418] Key type .llcrypt registered [ 6913.924188] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6914.179112] Key type .llcrypt unregistered [ 6914.181886] Key type ._llcrypt unregistered [ 6932.007421] Key type ._llcrypt registered [ 6932.008881] Key type .llcrypt registered [ 6932.073109] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6932.237506] Key type .llcrypt unregistered [ 6932.239312] Key type ._llcrypt unregistered [ 6949.940941] Key type ._llcrypt registered [ 6949.942542] Key type .llcrypt registered [ 6950.027595] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6950.244515] Key type .llcrypt unregistered [ 6950.246757] Key type ._llcrypt unregistered [ 6967.568605] Key type ._llcrypt registered [ 6967.570393] Key type .llcrypt registered [ 6967.640176] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6967.891043] Key type .llcrypt unregistered [ 6967.893666] Key type ._llcrypt unregistered [ 6984.188268] Key type ._llcrypt registered [ 6984.190185] Key type .llcrypt registered [ 6984.245254] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6984.395594] Key type .llcrypt unregistered [ 6984.397150] Key type ._llcrypt unregistered [ 6998.532148] Key type ._llcrypt registered [ 6998.533819] Key type .llcrypt registered [ 6998.589229] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6998.767349] Key type .llcrypt unregistered [ 6998.769476] Key type ._llcrypt unregistered [ 7025.449547] Key type ._llcrypt registered [ 7025.451083] Key type .llcrypt registered [ 7025.496761] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7025.622572] Key type .llcrypt unregistered [ 7025.623919] Key type ._llcrypt unregistered [ 7039.228164] Key type ._llcrypt registered [ 7039.229649] Key type .llcrypt registered [ 7039.281725] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7039.424242] Key type .llcrypt unregistered [ 7039.425651] Key type ._llcrypt unregistered [ 7054.752892] Key type ._llcrypt registered [ 7054.754448] Key type .llcrypt registered [ 7054.806922] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7054.989350] Key type .llcrypt unregistered [ 7054.990994] Key type ._llcrypt unregistered [ 7077.859279] Key type ._llcrypt registered [ 7077.860562] Key type .llcrypt registered [ 7077.907905] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7085.307584] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7085.697732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7085.739856] alg: No test for adler32 (adler32-zlib) [ 7086.618687] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 7086.731982] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7088.319134] Key type lgssc registered [ 7088.775286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7103.330141] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7103.336697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.433445] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7104.446723] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7104.488813] Lustre: tst250fs-MDT0000: new disk, initializing [ 7104.521108] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7104.529248] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7106.171309] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7111.184181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.238440] Lustre: 372121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7111.268776] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7111.271420] Lustre: Skipped 1 previous similar message [ 7111.308657] Lustre: tst250fs-MDT0001: new disk, initializing [ 7111.356238] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7111.372348] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7111.377292] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7113.105054] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7115.708509] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7118.613436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.744426] Lustre: tst250fs-OST0000: new disk, initializing [ 7118.747634] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7118.752262] Lustre: 373751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7118.795347] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7121.158818] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7121.162862] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7121.230482] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7121.370802] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7126.518208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.593400] Lustre: tst250fs-OST0001: new disk, initializing [ 7126.595571] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7126.599626] Lustre: 374620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7126.637778] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7128.866577] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7131.156025] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7131.160506] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7131.200363] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7131.818323] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7146.469690] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.471247] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7146.477664] Lustre: Skipped 3 previous similar messages [ 7146.479916] Lustre: Skipped 2 previous similar messages [ 7149.227704] Lustre: server umount tst250fs-MDT0000 complete [ 7150.805277] LustreError: 372112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786471236 with bad export cookie 17468354303784414755 [ 7150.810967] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7150.811556] LustreError: 372112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7150.964049] Lustre: server umount tst250fs-MDT0001 complete [ 7162.902102] Lustre: server umount tst250fs-OST0000 complete [ 7174.881493] Lustre: server umount tst250fs-OST0001 complete [ 7181.815953] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7184.978754] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7197.020842] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:01:21 (1786471281) [ 7197.670125] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7204.195919] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7208.273867] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7221.256719] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7064 sec ========== 14:01:45 (1786471305) [ 7221.994575] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:01:46 (1786471306) === [ 7223.530871] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:01:48 (1786471308) === [ 7239.051891] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7240.352377] Key type lgssc unregistered [ 7240.502648] LNet: 383743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7240.507925] LNetError: 383743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7240.519666] LNet: Removed LNI 192.168.204.146@tcp [ 7240.933249] Key type .llcrypt unregistered [ 7240.935639] Key type ._llcrypt unregistered