[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 542975917 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002343] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007330] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010115] LSM: Security Framework initializing [ 0.011037] Yama: becoming mindful. [ 0.012025] SELinux: Initializing. [ 0.013049] *** VALIDATE selinux *** [ 0.020704] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024344] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026008] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] *** VALIDATE tmpfs *** [ 0.030238] *** VALIDATE proc *** [ 0.031179] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.033234] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036022] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040372] debug: unmapping init [mem 0xffffffff84059000-0xffffffff84060fff] [ 0.042237] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043548] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044018] ... version: 2 [ 0.045008] ... bit width: 48 [ 0.046006] ... generic registers: 4 [ 0.047008] ... value mask: 0000ffffffffffff [ 0.048007] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051261] rcu: Hierarchical SRCU implementation. [ 0.053404] smp: Bringing up secondary CPUs ... [ 0.054523] x86: Booting SMP configuration: [ 0.055012] .... node #0, CPUs: #1 #2 #3 [ 0.065059] smp: Brought up 1 node, 4 CPUs [ 0.067010] smpboot: Max logical packages: 1 [ 0.068016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.108019] node 0 deferred pages initialised in 38ms [ 0.112113] devtmpfs: initialized [ 0.114313] x86/mm: Memory block size: 128MB [ 0.117827] gcov: version magic: 0x41383552 [ 0.121358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.123059] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.127508] pinctrl core: initialized pinctrl subsystem [ 0.130181] [ 0.131007] ************************************************************* [ 0.134010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137009] ** ** [ 0.140011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.143139] ** ** [ 0.145009] ** This means that this kernel is built to expose internal ** [ 0.148010] ** IOMMU data structures, which may compromise security on ** [ 0.151010] ** your system. ** [ 0.153007] ** ** [ 0.156009] ** If you see this message and you are not debugging the ** [ 0.158012] ** kernel, report this immediately to your vendor! ** [ 0.160009] ** ** [ 0.163008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165010] ************************************************************* [ 0.168857] NET: Registered protocol family 16 [ 0.170627] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180101] cpuidle: using governor menu [ 0.183092] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185516] PCI: Using configuration type 1 for base access [ 0.188220] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198336] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199066] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201048] cryptd: max_cpu_qlen set to 1000 [ 0.204262] ACPI: Added _OSI(Module Device) [ 0.205000] ACPI: Added _OSI(Processor Device) [ 0.206013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213423] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221622] ACPI: Interpreter enabled [ 0.223052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225008] ACPI: Using IOAPIC for interrupt routing [ 0.226154] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231475] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.243737] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.246106] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258714] acpiphp: Slot [2] registered [ 0.260132] acpiphp: Slot [5] registered [ 0.262100] acpiphp: Slot [6] registered [ 0.264176] acpiphp: Slot [7] registered [ 0.266179] acpiphp: Slot [8] registered [ 0.268121] acpiphp: Slot [9] registered [ 0.270176] acpiphp: Slot [10] registered [ 0.272106] acpiphp: Slot [3] registered [ 0.274135] acpiphp: Slot [4] registered [ 0.276077] acpiphp: Slot [11] registered [ 0.277159] acpiphp: Slot [12] registered [ 0.279082] acpiphp: Slot [13] registered [ 0.280072] acpiphp: Slot [14] registered [ 0.281080] acpiphp: Slot [15] registered [ 0.283127] acpiphp: Slot [16] registered [ 0.284089] acpiphp: Slot [17] registered [ 0.286069] acpiphp: Slot [18] registered [ 0.287063] acpiphp: Slot [19] registered [ 0.289100] acpiphp: Slot [20] registered [ 0.290088] acpiphp: Slot [21] registered [ 0.291246] acpiphp: Slot [22] registered [ 0.293089] acpiphp: Slot [23] registered [ 0.295078] acpiphp: Slot [24] registered [ 0.297083] acpiphp: Slot [25] registered [ 0.298069] acpiphp: Slot [26] registered [ 0.300083] acpiphp: Slot [27] registered [ 0.301253] acpiphp: Slot [28] registered [ 0.303079] acpiphp: Slot [29] registered [ 0.305078] acpiphp: Slot [30] registered [ 0.306069] acpiphp: Slot [31] registered [ 0.308066] PCI host bridge to bus 0000:00 [ 0.309012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324269] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326869] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330099] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.355017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358405] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361859] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373049] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.387011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.393012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397597] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.476287] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.497011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.526014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.561012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.589509] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.604015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.615020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.638012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.649000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.658013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.666013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.683015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.694841] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.710080] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.727020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.752025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.774269] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.783012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.793013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.819013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.840088] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.845601] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.847458] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.851405] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.859254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.865105] iommu: Default domain type: Passthrough [ 0.867357] SCSI subsystem initialized [ 0.870137] ACPI: bus type USB registered [ 0.871071] usbcore: registered new interface driver usbfs [ 0.873059] usbcore: registered new interface driver hub [ 0.875052] usbcore: registered new device driver usb [ 0.877158] pps_core: LinuxPPS API ver. 1 registered [ 0.879008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.884049] PTP clock support registered [ 0.887143] EDAC MC: Ver: 3.0.0 [ 0.890104] PCI: Using ACPI for IRQ routing [ 0.893221] NetLabel: Initializing [ 0.895007] NetLabel: domain hash size = 128 [ 0.897008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.900069] NetLabel: unlabeled traffic allowed by default [ 0.903360] vgaarb: loaded [ 0.905238] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.906006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.914769] clocksource: Switched to clocksource kvm-clock [ 1.077451] VFS: Disk quotas dquot_6.6.0 [ 1.079114] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.083279] *** VALIDATE ramfs *** [ 1.084592] *** VALIDATE hugetlbfs *** [ 1.089988] pnp: PnP ACPI init [ 1.092391] pnp: PnP ACPI: found 6 devices [ 1.115934] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.119550] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.121411] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.123224] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.125437] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.128208] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.130989] NET: Registered protocol family 2 [ 1.133685] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.138527] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.143363] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.152108] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.157704] TCP: Hash tables configured (established 65536 bind 65536) [ 1.162502] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.167962] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.172333] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.176081] NET: Registered protocol family 1 [ 1.180039] RPC: Registered named UNIX socket transport module. [ 1.182889] RPC: Registered udp transport module. [ 1.185379] RPC: Registered tcp transport module. [ 1.187307] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.189703] NET: Registered protocol family 44 [ 1.191247] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.193242] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.195278] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.197603] PCI: CLS 0 bytes, default 64 [ 1.199847] Unpacking initramfs... [ 3.089391] debug: unmapping init [mem 0xffff89597cc54000-0xffff89597ffbffff] [ 3.093841] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.095838] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.098637] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.589521] Initialise system trusted keyrings [ 3.591120] Key type blacklist registered [ 3.594072] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.611025] zbud: loaded [ 3.618508] *** VALIDATE nfs *** [ 3.620018] *** VALIDATE nfs4 *** [ 3.623212] pstore: using deflate compression [ 3.628248] Platform Keyring initialized [ 3.759978] NET: Registered protocol family 38 [ 3.762850] Key type asymmetric registered [ 3.767996] Asymmetric key parser 'x509' registered [ 3.770390] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.773961] io scheduler mq-deadline registered [ 3.775859] io scheduler kyber registered [ 3.777955] io scheduler bfq registered [ 3.780375] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.784830] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.791612] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.795957] ACPI: Power Button [PWRF] [ 3.803530] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.815615] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.842324] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.853419] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.871579] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.900412] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.937788] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.946054] Non-volatile memory driver v1.3 [ 3.949360] Linux agpgart interface v0.103 [ 4.000066] virtio_blk virtio1: [vda] 150504 512-byte logical blocks (77.1 MB/73.5 MiB) [ 4.003725] vda: detected capacity change from 0 to 77058048 [ 4.020603] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.024228] vdb: detected capacity change from 0 to 1073741824 [ 4.040233] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.044032] vdc: detected capacity change from 0 to 2621440000 [ 4.061163] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.064200] vdd: detected capacity change from 0 to 2621440000 [ 4.080858] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.083878] vde: detected capacity change from 0 to 4294967296 [ 4.100725] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.104063] vdf: detected capacity change from 0 to 4294967296 [ 4.111279] libphy: Fixed MDIO Bus: probed [ 4.118319] usbcore: registered new interface driver usbserial_generic [ 4.121167] usbserial: USB Serial support registered for generic [ 4.124280] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.129521] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.132262] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.135178] mousedev: PS/2 mouse device common for all mice [ 4.139025] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.140819] rtc_cmos 00:05: RTC can wake from S4 [ 4.150924] rtc_cmos 00:05: registered as rtc0 [ 4.153295] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.154199] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.154277] intel_pstate: CPU model not supported [ 4.156568] hid: raw HID events driver (C) Jiri Kosina [ 4.167407] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.168570] usbcore: registered new interface driver usbhid [ 4.177434] usbhid: USB HID core driver [ 4.179395] drop_monitor: Initializing network drop monitor service [ 4.181954] Initializing XFRM netlink socket [ 4.184373] NET: Registered protocol family 10 [ 4.187826] Segment Routing with IPv6 [ 4.189704] NET: Registered protocol family 17 [ 4.191830] mpls_gso: MPLS GSO support [ 4.196715] RAS: Correctable Errors collector initialized. [ 4.199163] AVX version of gcm_enc/dec engaged. [ 4.201235] AES CTR mode by8 optimization enabled [ 4.302513] sched_clock: Marking stable (4302379682, 0)->(5286913640, -984533958) [ 4.308885] registered taskstats version 1 [ 4.310954] Loading compiled-in X.509 certificates [ 4.313954] zswap: loaded using pool lzo/zbud [ 4.346408] Key type big_key registered [ 4.364879] Key type encrypted registered [ 4.366277] ima: No TPM chip found, activating TPM-bypass! [ 4.368664] ima: Allocated hash algorithm: sha1 [ 4.370500] ima: No architecture policies found [ 4.372454] evm: Initialising EVM extended attributes: [ 4.374464] evm: security.selinux [ 4.375715] evm: security.ima [ 4.376839] evm: security.capability [ 4.378331] evm: HMAC attrs: 0x1 [ 4.380597] rtc_cmos 00:05: setting system clock to 2026-09-08 20:50:37 UTC (1788900637) [ 4.388527] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 4.392346] debug: unmapping init [mem 0xffffffff83d82000-0xffffffff84058fff] [ 4.404144] Write protecting the kernel read-only data: 28672k [ 4.408721] debug: unmapping init [mem 0xffffffff82403000-0xffffffff825fffff] [ 4.414266] debug: unmapping init [mem 0xffffffff82d14000-0xffffffff82dfffff] [ 4.471777] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.481846] systemd[1]: Detected virtualization kvm. [ 4.484168] systemd[1]: Detected architecture x86-64. [ 4.486195] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.513278] systemd[1]: No hostname configured. [ 4.516783] systemd[1]: Set hostname to . [ 4.519294] random: systemd: uninitialized urandom read (16 bytes read) [ 4.521967] systemd[1]: Initializing machine ID from random generator. [ 4.607563] random: ln: uninitialized urandom read (6 bytes read) [ 4.785554] random: systemd: uninitialized urandom read (16 bytes read) [ 4.789710] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.802916] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.814963] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.990352] device-mapper: uevent: version 1.0.3 [ 5.992819] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.001504] random: fast init done Starting dracut initqueue hook... [ 7.021038] virtio_net virtio0 ens2: renamed from eth0 [ 7.038472] scsi host0: ata_piix [ 7.244789] scsi host1: ata_piix [ 7.248841] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.252385] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.898704] random: crng init done [ 11.900060] random: 7 urandom warning(s) missed due to ratelimiting [ 13.412334] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.755068] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.522034] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.146635] SELinux: Disabled at runtime. [ 17.220405] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.239758] systemd[1]: Detected virtualization kvm. [ 17.242019] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.981354] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.984965] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.994801] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.998694] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.002285] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.009509] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.021251] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ 18.278218] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.210341] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.978642] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.003748] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.278211] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.296489] EDAC sbridge: Ver: 1.1.2 [ 22.208511] Key type dns_resolver registered [ 22.614526] NFS: Registering the id_resolver key type [ 22.616236] Key type id_resolver registered [ 22.617861] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg444-server login: [ 45.958768] libcfs: loading out-of-tree module taints kernel. [ 45.974965] Key type ._llcrypt registered [ 45.976520] Key type .llcrypt registered [ 46.016526] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 53.479332] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 54.056819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.064071] alg: No test for adler32 (adler32-zlib) [ 55.042098] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 55.346174] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 56.952104] Key type lgssc registered [ 57.591299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 64.559643] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.057343] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 84.399785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 84.407169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.492445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 85.503729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 85.541829] Lustre: lustre-MDT0000: new disk, initializing [ 85.563467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.571739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 86.896923] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 92.041460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 92.074820] Lustre: 6481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 92.085787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 92.088519] Lustre: Skipped 1 previous similar message [ 92.127655] Lustre: lustre-MDT0001: new disk, initializing [ 92.145569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 92.153730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 92.157856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 93.442156] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.694411] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 98.731558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 98.821060] Lustre: lustre-OST0000: new disk, initializing [ 98.823387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 98.826605] Lustre: 8421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 98.845097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.618936] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.029462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 106.078407] Lustre: lustre-OST0001: new disk, initializing [ 106.079761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 106.081958] Lustre: 9481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 106.099988] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 107.313522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 107.317735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 107.322594] Lustre: Skipped 1 previous similar message [ 107.351484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 107.353898] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 108.116616] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.976573] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 118.484439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 120.452114] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing check_logdir /tmp/testlogs/ [ 121.852546] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing yml_node [ 123.251409] Lustre: DEBUG MARKER: Client: 2.17.58.5 [ 124.114553] Lustre: DEBUG MARKER: MDS: 2.17.58.5 [ 124.930597] Lustre: DEBUG MARKER: OSS: 2.17.58.5 [ 125.423070] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 16:52:38 EDT 2026 [ 130.786467] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 131.264750] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 143.329242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 143.329253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 143.329826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 143.329833] Lustre: Skipped 1 previous similar message [ 143.340326] Lustre: Skipped 3 previous similar messages [ 148.448786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.452628] Lustre: Skipped 5 previous similar messages [ 149.145944] Lustre: server umount lustre-MDT0000 complete [ 150.533738] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788900783 with bad export cookie 5953611902001794852 [ 150.535082] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 150.537430] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 153.568754] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 153.569560] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 153.575037] Lustre: Skipped 1 previous similar message [ 155.616512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 155.618005] Lustre: Skipped 2 previous similar messages [ 156.713903] Lustre: server umount lustre-MDT0001 complete [ 164.092908] Lustre: server umount lustre-OST0000 complete [ 171.403504] Lustre: server umount lustre-OST0001 complete [ 173.470280] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 176.641926] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 188.926955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.010400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.022608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.062546] Lustre: lustre-MDT0000: new disk, initializing [ 189.094368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.103171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.106714] Lustre: Skipped 1 previous similar message [ 190.545430] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 194.698458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.740553] Lustre: 15030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.850838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.854503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.229454] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 199.939512] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 201.752133] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 204.034785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.142791] Lustre: lustre-OST0000: new disk, initializing [ 204.144752] Lustre: Skipped 1 previous similar message [ 204.146996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.149499] Lustre: Skipped 2 previous similar messages [ 204.152113] Lustre: 16999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.182659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.185276] Lustre: Skipped 1 previous similar message [ 206.086124] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 206.139773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 206.142692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 206.191506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.773405] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 211.093111] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 211.165128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 212.500284] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 212.569954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 216.546694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 216.547616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 216.550304] LustreError: Skipped 1 previous similar message [ 216.552019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 216.555451] Lustre: Skipped 1 previous similar message [ 219.275997] Lustre: server umount lustre-OST0000 complete [ 220.640915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 220.643458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 226.584583] Lustre: server umount lustre-MDT0000 complete [ 226.785068] LustreError: 15040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 226.791115] LustreError: 15040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 227.834474] LustreError: 16998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788900860 with bad export cookie 5953611902001796294 [ 227.837056] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 227.837803] LustreError: 16998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 227.943978] Lustre: server umount lustre-MDT0001 complete [ 229.680090] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:54:22 (1788900862) [ 235.410180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.549838] LustreError: 19866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 235.573440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.939155] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 239.782867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.261996] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 243.651372] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 245.434428] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 246.147745] Lustre: Failing over lustre-MDT0000 [ 246.229058] Lustre: server umount lustre-MDT0000 complete [ 250.273548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.274242] LustreError: 19861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 250.277717] Lustre: Skipped 2 previous similar messages [ 250.283889] LustreError: 19861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 255.457032] LustreError: 19867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.462503] LustreError: 19867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 259.190935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 259.237440] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 260.809957] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 264.603838] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 264.673935] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 264.676372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 264.681578] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 269.792847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 269.795135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.799196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.801166] Lustre: Skipped 4 previous similar messages [ 272.794978] Lustre: server umount lustre-MDT0000 complete [ 274.011313] LustreError: 19848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788900907 with bad export cookie 5953611902001797708 [ 274.014579] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 274.014750] LustreError: 19848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 274.131053] Lustre: server umount lustre-MDT0001 complete [ 277.000294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 277.139081] LustreError: 23197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.160459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 277.162764] Lustre: Skipped 2 previous similar messages [ 278.418215] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 281.329100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 282.763664] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 284.999653] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 286.753261] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 287.448195] Lustre: Failing over lustre-MDT0000 [ 287.535228] Lustre: server umount lustre-MDT0000 complete [ 291.553580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.556980] Lustre: Skipped 1 previous similar message [ 291.558942] LustreError: 23197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.564787] LustreError: 23197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 300.493449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 300.532726] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.951967] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 305.617537] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 305.633096] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 305.635984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 305.639806] Lustre: Skipped 1 previous similar message [ 305.645649] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 307.610747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.612538] Lustre: Skipped 3 previous similar messages [ 313.748068] Lustre: server umount lustre-MDT0000 complete [ 314.960141] LustreError: 23933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788900948 with bad export cookie 5953611902001798646 [ 314.963215] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.963310] LustreError: 23933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 315.082042] Lustre: server umount lustre-MDT0001 complete [ 317.861976] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:55:50 (1788900950) [ 322.878639] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 325.708155] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 337.405938] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 340.870711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.965644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 340.978643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 341.010795] Lustre: lustre-MDT0000: new disk, initializing [ 341.035668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 342.334427] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 346.440791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.480188] Lustre: 29557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 346.567429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 346.570558] Lustre: Skipped 3 previous similar messages [ 346.582694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 347.862131] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 351.362092] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 353.059954] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 355.103441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 355.201525] Lustre: 31527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 356.946576] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 357.174146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 357.183667] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 357.206623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 357.212589] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 360.544498] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 362.417322] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 367.586487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.586664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 367.590132] Lustre: Skipped 3 previous similar messages [ 367.590794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 367.592652] LustreError: Skipped 1 previous similar message [ 367.593718] Lustre: Skipped 3 previous similar messages [ 371.874774] Lustre: server umount lustre-OST0000 complete [ 377.313234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 379.425559] Lustre: server umount lustre-MDT0000 complete [ 380.663595] LustreError: 30353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788901013 with bad export cookie 5953611902001799052 [ 380.667284] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.667693] LustreError: 30353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 380.843787] Lustre: server umount lustre-MDT0001 complete [ 384.275564] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 385.295851] Key type lgssc unregistered [ 385.427401] LNet: 33477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.430381] LNetError: 33477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 385.442315] LNet: Removed LNI 192.168.204.144@tcp [ 385.771094] Key type .llcrypt unregistered [ 385.772097] Key type ._llcrypt unregistered [ 394.675153] Key type ._llcrypt registered [ 394.676179] Key type .llcrypt registered [ 394.712759] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 395.030591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 395.087854] alg: No test for adler32 (adler32-zlib) [ 395.943595] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 396.028839] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 397.616156] Key type lgssc registered [ 397.976985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 400.542449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 400.548832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.696476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.085509] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 405.967982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.102324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.453142] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 409.922278] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 411.738448] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 413.984524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.103816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.989138] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 416.167976] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 418.431306] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 433.514833] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:57:46 (1788901066) [ 434.134248] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 434.727973] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 436.778889] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:57:49 (1788901069) [ 439.424156] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:57:52 (1788901072) [ 439.758816] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 442.393564] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 442.395424] Lustre: Skipped 2 previous similar messages [ 442.953734] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:57:55 (1788901075) [ 449.565261] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 449.567060] Lustre: Skipped 10 previous similar messages [ 450.386376] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:58:03 (1788901083) [ 451.261276] Lustre: *** cfs_fail_loc=131b, val=2*** [ 452.403969] Lustre: *** cfs_fail_loc=131b, val=2*** [ 455.101190] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:58:08 (1788901088) [ 459.226289] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 459.228450] Lustre: Skipped 7 previous similar messages [ 459.759636] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:58:12 (1788901092) [ 467.425383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 467.428407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.434090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.450514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.450685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.451277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 469.581794] Lustre: server umount lustre-OST0000 complete [ 472.544820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 472.546878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.550557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.552351] Lustre: Skipped 1 previous similar message [ 478.688969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 478.691072] Lustre: Skipped 3 previous similar messages [ 483.809075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 483.811659] Lustre: Skipped 1 previous similar message [ 485.344121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 485.402514] Lustre: server umount lustre-MDT0000 complete [ 486.610125] LustreError: 34843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788901119 with bad export cookie 13718064163097182565 [ 486.611833] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.614401] LustreError: 34843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 486.722062] Lustre: server umount lustre-MDT0001 complete [ 490.194344] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 491.256447] Key type lgssc unregistered [ 491.382325] LNet: 42712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 491.384630] LNetError: 42712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 491.395293] LNet: Removed LNI 192.168.204.144@tcp [ 491.683351] Key type .llcrypt unregistered [ 491.684743] Key type ._llcrypt unregistered [ 501.571490] Key type ._llcrypt registered [ 501.573843] Key type .llcrypt registered [ 501.683778] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 513.671957] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 514.852128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 514.891976] alg: No test for adler32 (adler32-zlib) [ 515.921400] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 516.133549] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 517.784312] Key type lgssc registered [ 519.063506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 548.585194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 548.608738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.769370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 549.788099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 549.873835] Lustre: lustre-MDT0000: new disk, initializing [ 549.929508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.940753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 554.666830] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 564.757927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.839447] Lustre: 45840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 564.920241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 564.923723] Lustre: Skipped 1 previous similar message [ 564.981293] Lustre: lustre-MDT0001: new disk, initializing [ 565.076254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 565.135625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 565.157443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 569.125898] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 577.517647] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 583.146605] hrtimer: interrupt took 3950589 ns [ 583.460755] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 589.288771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 589.560744] Lustre: lustre-OST0000: new disk, initializing [ 589.564461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 589.571327] Lustre: 47806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 589.661169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.475311] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 599.063334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 599.071522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 599.143238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 600.326916] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 602.752140] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 602.863661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 605.606887] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 605.730189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 609.249821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.252849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.254247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 609.256871] Lustre: Skipped 1 previous similar message [ 609.267252] Lustre: Skipped 1 previous similar message [ 614.372161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 614.376395] Lustre: Skipped 1 previous similar message [ 619.495511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.499602] Lustre: Skipped 1 previous similar message [ 621.027039] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 621.120764] Lustre: server umount lustre-OST0000 complete [ 624.610347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.611546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 624.619972] Lustre: Skipped 1 previous similar message [ 624.626684] Lustre: Skipped 1 previous similar message [ 629.489572] Lustre: server umount lustre-MDT0000 complete [ 631.935248] LustreError: 45831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788901265 with bad export cookie 15495080998078206611 [ 631.937621] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.943444] LustreError: 45831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 632.146926] Lustre: server umount lustre-MDT0001 complete [ 638.538075] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:01:10 (1788901270) [ 642.688607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.973506] LustreError: 50278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.987603] LustreError: 50278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 643.042074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.825349] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 648.163132] LustreError: 50279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 648.186276] LustreError: 50279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 651.490983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.760894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 654.834898] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 659.888322] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 663.399404] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 667.764944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 667.993474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 671.759497] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 676.181611] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 679.794197] LustreError: 52807:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 680.330219] Lustre: Modifying parameter general.timeout=1 in log params [ 681.443268] Lustre: Modifying parameter general.timeout=3 in log params [ 681.448767] Lustre: Skipped 1 previous similar message [ 683.514363] Lustre: Modifying parameter general.timeout=7 in log params [ 683.517426] Lustre: Skipped 3 previous similar messages [ 687.598275] Lustre: Modifying parameter general.timeout=15 in log params [ 687.601535] Lustre: Skipped 7 previous similar messages [ 695.871824] Lustre: Modifying parameter general.timeout=31 in log params [ 695.874363] Lustre: Skipped 15 previous similar messages [ 712.139089] Lustre: Modifying parameter general.timeout=60 in log params [ 712.148613] Lustre: Skipped 28 previous similar messages [ 744.316511] Lustre: Modifying parameter general.timeout=117 in log params [ 744.319700] Lustre: Skipped 56 previous similar messages [ 759.803857] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:03:12 (1788901392) [ 761.969312] Lustre: 59719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 763.373383] Lustre: 59817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 766.108587] Lustre: 60029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 766.117426] Lustre: 60029:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 785.295083] Lustre: 61582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 785.301784] Lustre: 61582:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 785.304958] LustreError: 61582:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 788.076742] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:03:40 (1788901420) [ 800.228177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.233339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 800.237658] Lustre: Skipped 2 previous similar messages [ 800.238706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 800.243913] Lustre: Skipped 2 previous similar messages [ 800.246286] LustreError: Skipped 1 previous similar message [ 804.112615] Lustre: server umount lustre-MDT0000 complete [ 805.350264] LustreError: 50279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 805.374923] LustreError: 50279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 807.042619] LustreError: 50258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788901440 with bad export cookie 15495080998078207500 [ 807.056954] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.058389] LustreError: 50258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 807.358766] Lustre: server umount lustre-MDT0001 complete [ 821.569504] Lustre: server umount lustre-OST0000 complete [ 844.388067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.611266] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 844.632632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 844.638662] Lustre: Skipped 35 previous similar messages [ 844.896623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.710594] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 855.960915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.056420] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 856.085455] Lustre: 64115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 859.376868] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 865.479077] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 869.976217] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 875.526094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.677780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 880.669609] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 886.155815] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 887.517810] Lustre: 66678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 897.506316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.512113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.520606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.084325] Lustre: server umount lustre-OST0000 complete [ 902.625655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 902.634094] LustreError: Skipped 1 previous similar message [ 902.640474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.660396] Lustre: Skipped 1 previous similar message [ 908.456057] Lustre: server umount lustre-MDT0000 complete [ 911.254163] LustreError: 64106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788901544 with bad export cookie 15495080998078210111 [ 911.264503] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 911.265034] LustreError: 64106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 911.572909] Lustre: server umount lustre-MDT0001 complete [ 919.446173] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 921.452733] Key type lgssc unregistered [ 921.693309] LNet: 67890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.700536] LNetError: 67890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.713406] LNet: Removed LNI 192.168.204.144@tcp [ 922.337604] Key type .llcrypt unregistered [ 922.339625] Key type ._llcrypt unregistered [ 940.655510] Key type ._llcrypt registered [ 940.658366] Key type .llcrypt registered [ 940.749467] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 941.607650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 941.637774] alg: No test for adler32 (adler32-zlib) [ 942.652263] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 942.874131] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 944.528338] Key type lgssc registered [ 945.403507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.081387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 951.096052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.369145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.515810] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 961.921281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.478755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.139817] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 971.565433] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 975.852658] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 981.412514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.880429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.412705] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 991.860932] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1008.161962] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1018.849601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1018.853076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.855235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1018.879932] Lustre: Skipped 2 previous similar messages [ 1023.292176] Lustre: server umount lustre-MDT0000 complete [ 1023.977114] LustreError: 69273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.002856] LustreError: 69273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1026.268397] LustreError: 69259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788901659 with bad export cookie 1538704418204546007 [ 1026.279415] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.280155] LustreError: 69259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1026.624696] Lustre: server umount lustre-MDT0001 complete [ 1040.364887] Lustre: server umount lustre-OST0000 complete [ 1061.267597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.578951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1061.606729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1061.894711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.226748] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1071.524335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.596298] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1071.617054] Lustre: 74171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1071.971229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.911106] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1079.646031] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1084.026766] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1089.393367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.523706] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1089.846549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.535137] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1099.754272] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1100.912979] Lustre: 76742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1102.266607] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1108.450537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1108.460167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.471471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.476757] Lustre: Skipped 2 previous similar messages [ 1111.807077] Lustre: server umount lustre-OST0000 complete [ 1118.177669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1118.186835] LustreError: Skipped 1 previous similar message [ 1118.190184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.197069] Lustre: Skipped 1 previous similar message [ 1118.202470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1118.208578] Lustre: Skipped 1 previous similar message [ 1120.561555] Lustre: server umount lustre-MDT0000 complete [ 1123.252854] LustreError: 76744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788901756 with bad export cookie 1538704418204546854 [ 1123.253648] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1123.262351] LustreError: 76744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1123.572062] Lustre: server umount lustre-MDT0001 complete [ 1130.831362] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1132.974704] Key type lgssc unregistered [ 1133.208612] LNet: 77948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.215961] LNetError: 77948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.228706] LNet: Removed LNI 192.168.204.144@tcp [ 1133.821518] Key type .llcrypt unregistered [ 1133.825353] Key type ._llcrypt unregistered [ 1150.136761] Key type ._llcrypt registered [ 1150.139385] Key type .llcrypt registered [ 1150.214686] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1150.874321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1150.897911] alg: No test for adler32 (adler32-zlib) [ 1151.896548] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 1152.073579] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1153.712159] Key type lgssc registered [ 1154.654739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1160.264880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1160.276596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.601319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.829273] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1170.796793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.916314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1171.216628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1174.295848] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1179.253025] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1182.955046] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1187.679922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1188.069725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.385232] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1197.785094] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1212.917550] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1213.542273] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1214.802753] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1214.807331] Lustre: Skipped 1 previous similar message [ 1217.372673] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1217.379202] Lustre: Skipped 3 previous similar messages [ 1221.486500] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1221.490892] Lustre: Skipped 6 previous similar messages [ 1229.608516] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1229.612482] Lustre: Skipped 13 previous similar messages [ 1245.636838] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1245.645464] Lustre: Skipped 28 previous similar messages [ 1274.296617] Lustre: 87431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1275.945871] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:11:48 (1788901908) [ 1276.986109] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1278.223662] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:11:50 (1788901910) [ 1283.556042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.557702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1283.566499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1283.581392] Lustre: Skipped 1 previous similar message [ 1286.715150] Lustre: server umount lustre-OST0000 complete [ 1293.795038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1293.796367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.799661] LustreError: Skipped 1 previous similar message [ 1293.801441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.817430] Lustre: Skipped 2 previous similar messages [ 1295.174406] Lustre: server umount lustre-MDT0000 complete [ 1297.754977] LustreError: 80075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788901930 with bad export cookie 7347949816106816182 [ 1297.756277] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.766718] LustreError: 80075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1298.019570] Lustre: server umount lustre-MDT0001 complete [ 1304.792547] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1306.564412] Key type lgssc unregistered [ 1306.779190] LNet: 88679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.784840] LNetError: 88679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.795302] LNet: Removed LNI 192.168.204.144@tcp [ 1307.365816] Key type .llcrypt unregistered [ 1307.367467] Key type ._llcrypt unregistered [ 1310.151459] Key type ._llcrypt registered [ 1310.154668] Key type .llcrypt registered [ 1310.231320] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1313.309612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules [ 1313.991269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1314.118196] alg: No test for adler32 (adler32-zlib) [ 1315.099311] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 1315.280032] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1316.920170] Key type lgssc registered [ 1317.839778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1318.547649] LustreError: 89877:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1330.871990] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1336.841734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1340.067709] LustreError: 89877:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1340.104668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.487533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.481725] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1350.247198] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:13:02 (1788901982) [ 1351.650414] LustreError: 90901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.661561] LustreError: 90901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1353.327236] LustreError: 90932:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1353.334532] LustreError: 90932:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1353.340607] LustreError: 90932:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 1355.592829] Lustre: server umount lustre-MDT0000 complete [ 1363.838916] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1365.630176] Key type lgssc unregistered [ 1365.827716] LNet: 92425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.835390] LNetError: 92425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.848951] LNet: Removed LNI 192.168.204.144@tcp [ 1366.429632] Key type .llcrypt unregistered [ 1366.432322] Key type ._llcrypt unregistered [ 1377.535956] Key type ._llcrypt registered [ 1377.539217] Key type .llcrypt registered [ 1377.621153] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1378.172948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1378.247228] alg: No test for adler32 (adler32-zlib) [ 1379.203763] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 1379.363031] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1380.976283] Key type lgssc registered [ 1381.631840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1386.212735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1386.223605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.531848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.392185] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1396.396366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.958205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.391846] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1406.030266] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1410.145404] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1414.638915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1415.023617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.513699] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1424.835261] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1467.232338] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:14:59 (1788902099) [ 1468.271375] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1469.289431] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:15:01 (1788902101) [ 1473.506718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1473.511748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.519834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.533319] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.534587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.546860] Lustre: Skipped 2 previous similar messages [ 1477.582801] Lustre: server umount lustre-MDT0000 complete [ 1479.645746] LustreError: 94358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788902112 with bad export cookie 17003517779074893442 [ 1479.651888] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1479.652562] LustreError: 93614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.652575] LustreError: 93614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1479.654523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.654530] Lustre: Skipped 1 previous similar message [ 1479.655391] LustreError: 94358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1479.656629] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1479.829494] Lustre: server umount lustre-MDT0001 complete [ 1488.079842] Lustre: server umount lustre-OST0000 complete [ 1492.356562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.488224] LustreError: 97911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.530923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.781269] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1496.735545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.951362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1498.269060] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1500.514632] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1502.227962] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1507.081158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.175803] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1507.178285] LustreError: 97899:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1507.194045] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1507.197182] LustreError: 99993:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 1507.199280] LustreError: 99993:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 1507.201050] LustreError: 99993:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1507.229495] Lustre: server umount lustre-OST0000 complete [ 1507.230722] LustreError: 99993:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1510.543271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1510.641582] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1510.645385] LustreError: 97901:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1510.663703] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1510.667321] LustreError: 100458:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 1510.669590] LustreError: 100458:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 1510.671584] LustreError: 100458:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1510.706112] Lustre: server umount lustre-OST0000 complete [ 1510.708982] LustreError: 100458:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1514.847486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1514.945981] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1514.947992] Lustre: Found index 0 for lustre-OST0000, updating log [ 1514.950284] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1514.984479] Lustre: lustre-OST0000: new disk, initializing [ 1514.987203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1514.990967] Lustre: 100993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1515.084975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.588226] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1519.059824] Lustre: Failing over lustre-OST0000 [ 1519.160661] Lustre: server umount lustre-OST0000 complete [ 1521.954755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.955844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.962819] Lustre: Skipped 1 previous similar message [ 1521.969669] Lustre: Skipped 1 previous similar message [ 1527.266156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.272479] Lustre: Skipped 1 previous similar message [ 1531.393603] Lustre: server umount lustre-MDT0000 complete [ 1532.385824] LustreError: 97911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.391048] LustreError: 97911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1532.700922] LustreError: 100992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788902165 with bad export cookie 17003517779074896186 [ 1532.704201] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.708464] LustreError: 100992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1532.847410] Lustre: server umount lustre-MDT0001 complete [ 1536.340680] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:16:09 (1788902169) [ 1542.781212] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1546.890716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.048821] LustreError: 103330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.077138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.726192] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1551.889181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.024682] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1555.466776] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1558.819051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1559.112510] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1561.718860] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1562.217771] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1562.220180] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 1562.224025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1565.405902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1565.439299] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1565.477115] Lustre: lustre-OST0001: new disk, initializing [ 1565.479093] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1565.482153] Lustre: 105647:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1565.588448] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 1565.596326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1565.927206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1565.932873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1565.978166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1567.894129] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1569.021620] Lustre: lustre-OST0000: Denying connection for new client 45c767e9-981c-403a-a457-c6a3c8d3e095 (at 192.168.204.44@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 1571.298739] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 1571.303530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1571.561385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1572.335734] Lustre: lustre-OST0000: Recovery over after 0:10, of 1 clients 1 recovered and 0 were evicted. [ 1572.335873] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1572.339221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1579.255074] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1582.566904] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.568682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1582.569805] Lustre: Skipped 1 previous similar message [ 1586.781418] Lustre: server umount lustre-MDT0001 complete [ 1587.685479] LustreError: 103325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1587.690437] LustreError: 103325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1589.998197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.033425] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1590.035421] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1590.052474] Lustre: 103319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1590.324163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1590.327675] Lustre: Skipped 3 previous similar messages [ 1591.592538] LustreError: 107352:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1591.595311] LustreError: 107352:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1591.599218] LustreError: 103323:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1591.628501] LustreError: 107367:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.144@tcp: cfg command failed: rc = -17 [ 1591.631229] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1591.631229] [ 1591.773614] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1594.929217] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:17:07 (1788902227) [ 1595.372916] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1600.782318] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1607.077820] Lustre: 109254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1615.481797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1616.729986] Lustre: 110388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1667.041803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1667.041863] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.042562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.042572] Lustre: Skipped 3 previous similar messages [ 1667.047055] LustreError: Skipped 1 previous similar message [ 1667.054900] Lustre: Skipped 4 previous similar messages [ 1669.295795] Lustre: server umount lustre-MDT0000 complete [ 1670.645187] LustreError: 104070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788902303 with bad export cookie 17003517779074897131 [ 1670.646734] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.649496] LustreError: 104070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1670.771437] Lustre: server umount lustre-MDT0001 complete [ 1682.404275] Lustre: server umount lustre-OST0000 complete [ 1693.081797] Lustre: server umount lustre-OST0001 complete [ 1694.881646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1698.480325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1702.082764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1706.164223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1713.767606] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1717.535636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1717.547835] Lustre: lustre-MDT0000: reset Object Index mappings [ 1717.720795] LustreError: 114309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.728156] LustreError: 114309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1717.743572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.203317] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1722.256097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.262508] Lustre: lustre-MDT0001: reset Object Index mappings [ 1724.243323] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1725.349344] Lustre: 115457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1727.767907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1730.353793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1732.143690] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1733.664914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1736.230871] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1739.249548] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1742.319659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 1742.320801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1744.918402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1746.170028] Lustre: 117334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1762.202825] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:19:54 (1788902394) [ 1793.511779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.513402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.515097] Lustre: Skipped 2 previous similar messages [ 1793.517184] Lustre: Skipped 4 previous similar messages [ 1797.422628] Lustre: server umount lustre-MDT0000 complete [ 1798.569738] LustreError: 114290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788902431 with bad export cookie 17003517779074944129 [ 1798.572976] LustreError: 114290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1798.573314] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1798.626053] LustreError: 114306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1798.630296] LustreError: 114306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1804.983505] Lustre: server umount lustre-MDT0001 complete [ 1812.222441] Lustre: server umount lustre-OST0000 complete [ 1819.481735] Lustre: server umount lustre-OST0001 complete [ 1821.514593] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 1824.719600] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1836.347388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.434399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1836.445336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1836.481882] Lustre: lustre-MDT0000: new disk, initializing [ 1836.506189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1837.796348] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1840.605702] Lustre: server umount lustre-MDT0000 complete [ 1843.670510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.784835] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1843.791093] Lustre: 121590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1843.798261] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1845.265304] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1847.015910] Lustre: server umount lustre-MDT0000 complete [ 1850.268828] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:21:23 (1788902483) [ 1850.932020] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1851.526521] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:21:24 (1788902484) [ 1858.232556] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1862.354357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.502215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.504578] Lustre: Skipped 5 previous similar messages [ 1863.889254] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1866.924139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.982373] Lustre: 123639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1867.007138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1867.009126] Lustre: Skipped 1 previous similar message [ 1867.050806] Lustre: lustre-MDT0001: new disk, initializing [ 1867.097881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1867.102440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1868.445330] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1870.868365] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1873.173911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1873.276764] Lustre: lustre-OST0000: new disk, initializing [ 1873.278817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1873.281365] Lustre: 125216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1874.361053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1874.363939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1874.402657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1875.307212] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1879.534056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.594118] Lustre: 126087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1880.966756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1880.969504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1881.011437] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1881.508656] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1885.766027] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1886.893651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2031.774212] Lustre: 132463:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2031.777650] Lustre: 132463:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2031.781119] Lustre: 132463:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2031.784031] Lustre: 132463:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2031.788142] Lustre: 132463:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2031.790766] Lustre: 132463:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2135.298336] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:26:07 (1788902767) [ 2147.304397] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.305870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2147.312804] Lustre: Skipped 4 previous similar messages [ 2147.319894] Lustre: Skipped 9 previous similar messages [ 2149.135691] Lustre: server umount lustre-MDT0000 complete [ 2150.444228] LustreError: 126848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788902783 with bad export cookie 17003517779074998008 [ 2150.450451] LustreError: 126848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2150.453142] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2152.417852] LustreError: 126884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.428241] LustreError: 126884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2152.700295] Lustre: server umount lustre-MDT0001 complete [ 2160.472187] Lustre: server umount lustre-OST0000 complete [ 2167.959678] Lustre: server umount lustre-OST0001 complete [ 2170.408475] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2174.225408] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2188.351834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.500625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2188.522485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2188.525857] Lustre: Skipped 1 previous similar message [ 2188.587942] Lustre: lustre-MDT0000: new disk, initializing [ 2188.590420] Lustre: Skipped 1 previous similar message [ 2188.624703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.628850] Lustre: Skipped 3 previous similar messages [ 2188.637820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2190.634540] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2196.510927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.612871] Lustre: 135479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2196.849430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2199.512238] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2205.439859] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2208.847507] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2212.619310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2212.793842] Lustre: lustre-OST0000: new disk, initializing [ 2212.795822] Lustre: Skipped 1 previous similar message [ 2212.798500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2212.804368] Lustre: Skipped 2 previous similar messages [ 2212.807722] Lustre: 137444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2214.611093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2214.617771] Lustre: Skipped 1 previous similar message [ 2214.621761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2214.690772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2216.049480] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2221.765884] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2227.117670] Lustre: lustre-MDD0000: changelog on [ 2228.628477] Lustre: lustre-MDD0001: changelog on [ 3308.335596] Lustre: lustre-MDD0001: changelog off [ 3308.987763] Lustre: 135486:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3309.301747] Lustre: lustre-MDD0000: changelog off [ 3313.634671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.635411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3313.639568] Lustre: Skipped 5 previous similar messages [ 3313.641065] Lustre: Skipped 2 previous similar messages [ 3317.011102] Lustre: server umount lustre-OST0000 complete [ 3318.240753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3324.412374] Lustre: server umount lustre-MDT0000 complete [ 3325.612167] LustreError: 136271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903958 with bad export cookie 17003517779075802021 [ 3325.614074] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3325.616510] LustreError: 136271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3325.735114] Lustre: server umount lustre-MDT0001 complete [ 3329.345568] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3330.335137] Key type lgssc unregistered [ 3330.458337] LNet: 146288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3330.460805] LNetError: 146288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3330.468243] LNet: Removed LNI 192.168.204.144@tcp [ 3330.768110] Key type .llcrypt unregistered [ 3330.769059] Key type ._llcrypt unregistered [ 3336.960234] Key type ._llcrypt registered [ 3336.961968] Key type .llcrypt registered [ 3336.998953] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3343.334187] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3343.675871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3343.717977] alg: No test for adler32 (adler32-zlib) [ 3344.587391] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 3344.687258] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3346.272119] Key type lgssc registered [ 3346.622099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3357.434225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3357.440162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.514639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3358.524102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3358.556166] Lustre: lustre-MDT0000: new disk, initializing [ 3358.574864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.580189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3359.877731] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3363.918696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.959565] Lustre: 149457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3363.981165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3363.983133] Lustre: Skipped 1 previous similar message [ 3364.017996] Lustre: lustre-MDT0001: new disk, initializing [ 3364.049179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3364.062098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3364.066229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3365.387647] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3368.970098] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3370.733282] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3372.874433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.994441] Lustre: lustre-OST0000: new disk, initializing [ 3372.996760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3373.000089] Lustre: 151425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3373.031622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.850651] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3378.488938] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3381.237796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3381.242965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3381.265359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3385.784595] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_module obdecho/obdecho [ 3386.349657] LustreError: 152429:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3386.352487] LustreError: 152429:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3386.897502] LustreError: 152526:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3391.458329] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.459988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3391.462043] Lustre: Skipped 2 previous similar messages [ 3391.463818] Lustre: Skipped 1 previous similar message [ 3396.122930] Lustre: server umount lustre-MDT0000 complete [ 3397.360540] LustreError: 150253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904030 with bad export cookie 12641396400148914713 [ 3397.366712] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.480735] Lustre: server umount lustre-MDT0001 complete [ 3407.974282] Lustre: server umount lustre-OST0000 complete [ 3415.029399] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3418.609681] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3430.182674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.290029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3430.301243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3430.342707] Lustre: lustre-MDT0000: new disk, initializing [ 3430.367590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.373936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3431.746229] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3435.870448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.917108] Lustre: 156239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3435.946176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3435.949033] Lustre: Skipped 1 previous similar message [ 3435.985621] Lustre: lustre-MDT0001: new disk, initializing [ 3436.025082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3436.037961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3436.041040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3437.502204] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3441.206110] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3443.069415] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3445.252277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.350487] Lustre: lustre-OST0000: new disk, initializing [ 3445.352723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3445.355085] Lustre: 158208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3445.381925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.238778] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3448.892498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3448.895884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3448.933760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3450.913809] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3452.359761] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3452.443213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3453.732562] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3453.809793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3459.041937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.042643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.044716] LustreError: Skipped 1 previous similar message [ 3459.050054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3459.052784] Lustre: Skipped 2 previous similar messages [ 3460.495121] Lustre: server umount lustre-OST0000 complete [ 3464.161637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.162102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.165045] Lustre: Skipped 2 previous similar messages [ 3467.864151] Lustre: server umount lustre-MDT0000 complete [ 3469.056167] LustreError: 156232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904102 with bad export cookie 12641396400148915644 [ 3469.058824] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.060559] LustreError: 156232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3469.162111] Lustre: server umount lustre-MDT0001 complete [ 3470.352323] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:48:23 (1788904103) [ 3470.954816] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3471.596040] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:48:24 (1788904104) [ 3474.031334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.165109] LustreError: 160614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.171319] LustreError: 160614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3474.184532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.455522] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3478.143420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.526827] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3481.764665] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3483.412157] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3485.562536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3487.417350] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3489.695151] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3504.099249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.100275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3504.102790] Lustre: Skipped 2 previous similar messages [ 3504.104454] Lustre: Skipped 2 previous similar messages [ 3504.107258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3507.802496] Lustre: server umount lustre-MDT0000 complete [ 3508.995354] LustreError: 160596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904142 with bad export cookie 12641396400148916512 [ 3508.998263] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3508.999117] LustreError: 160596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3509.216917] LustreError: 162455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.217138] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.226798] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3509.228651] Lustre: Skipped 2 previous similar messages [ 3510.243192] LustreError: 160611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3513.248784] LustreError: 163106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.184671] Lustre: server umount lustre-MDT0001 complete [ 3517.262380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3521.501527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.594868] LustreError: 164476:0:(llog.c:1655:llog_backup()) MGC192.168.204.144@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3521.605366] Lustre: 164476:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.144@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3530.597609] LustreError: 164480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.851127] LustreError: 164480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.869464] LustreError: 164480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3556.326387] LustreError: 164482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.337703] LustreError: 164482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 3582.624429] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3582.626922] Lustre: 164476:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3590.113977] LustreError: 164482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.126843] LustreError: 164482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 3623.437645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3623.458272] Lustre: Skipped 2 previous similar messages [ 3628.420574] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3628.533919] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3643.572851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3650.042812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3654.182167] LustreError: 162446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.221945] LustreError: 162446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 48 previous similar messages [ 3659.802105] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3664.481443] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3675.222946] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3677.317404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3684.442842] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3686.513715] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3709.312150] Lustre: server umount lustre-MDT0000 complete [ 3710.951867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.959882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3710.968360] Lustre: Skipped 1 previous similar message [ 3713.275228] LustreError: 160640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904346 with bad export cookie 12641396400148922959 [ 3713.279219] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.287310] LustreError: 160640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3713.749864] Lustre: server umount lustre-MDT0001 complete [ 3729.652500] Lustre: server umount lustre-OST0000 complete [ 3750.804212] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3761.795501] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3801.562717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.754334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3801.778078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3801.848718] Lustre: lustre-MDT0000: new disk, initializing [ 3801.912187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.919934] Lustre: Skipped 1 previous similar message [ 3801.933315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3806.362611] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3817.215503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.323639] Lustre: 169585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3817.568554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3822.511170] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3831.771895] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3838.066201] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3845.082781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3845.399027] Lustre: lustre-OST0000: new disk, initializing [ 3845.404813] Lustre: Skipped 1 previous similar message [ 3845.413341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3845.422594] Lustre: Skipped 2 previous similar messages [ 3845.430648] Lustre: 171555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3851.899993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3851.916561] Lustre: Skipped 1 previous similar message [ 3851.925259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3852.056441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3852.414929] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3863.159080] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3869.147692] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3869.318554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3874.858703] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3875.060257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3877.859890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3877.863147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.870492] LustreError: Skipped 1 previous similar message [ 3877.871175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.871180] Lustre: Skipped 2 previous similar messages [ 3877.906137] Lustre: Skipped 2 previous similar messages [ 3883.386772] Lustre: server umount lustre-OST0000 complete [ 3894.000608] Lustre: server umount lustre-MDT0000 complete [ 3898.348179] LustreError: 171150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3898.361982] LustreError: 171150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3898.368715] LustreError: 169577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904531 with bad export cookie 12641396400148939787 [ 3898.380190] LustreError: 169577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3898.380643] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.858606] Lustre: server umount lustre-MDT0001 complete [ 3911.501946] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:55:42 (1788904542) [ 3920.280923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.894389] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3936.134295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.514264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.521425] Lustre: Skipped 3 previous similar messages [ 3941.434269] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3949.838956] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3956.591758] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3966.116261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3973.877603] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3983.491988] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3995.539889] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:57:06 (1788904626) [ 4001.763786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4001.764296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.776267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4001.776274] Lustre: Skipped 7 previous similar messages [ 4001.801071] Lustre: Skipped 3 previous similar messages [ 4005.949707] Lustre: server umount lustre-OST0000 complete [ 4016.645480] Lustre: server umount lustre-MDT0000 complete [ 4020.926849] LustreError: 173950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904654 with bad export cookie 12641396400148940669 [ 4020.929150] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4020.942421] LustreError: 173950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4021.355443] Lustre: server umount lustre-MDT0001 complete [ 4033.999500] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4037.010953] Key type lgssc unregistered [ 4037.412619] LNet: 177700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4037.418974] LNetError: 177700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4037.433780] LNet: Removed LNI 192.168.204.144@tcp [ 4038.361788] Key type .llcrypt unregistered [ 4038.364144] Key type ._llcrypt unregistered [ 4061.625145] Key type ._llcrypt registered [ 4061.626827] Key type .llcrypt registered [ 4061.775267] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4063.048265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4063.058629] alg: No test for adler32 (adler32-zlib) [ 4064.242718] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 4064.486865] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4066.192215] Key type lgssc registered [ 4067.703877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4078.246778] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4078.264453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4093.793592] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4098.912501] LustreError: 178944:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 4104.032452] LustreError: 178944:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4104.052802] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4104.060662] LustreError: MGC192.168.204.144@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4104.074694] LustreError: 178944:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4104.079108] LustreError: 178944:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 4104.086635] LustreError: 178944:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 4104.136709] Lustre: server umount lustre-OST0000 complete [ 4104.142831] LustreError: 178944:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4111.928207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.215166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.670086] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4121.896047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.215124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4125.400823] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4130.507393] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4134.273179] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4139.012153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4139.133672] LustreError: 181259:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4139.147574] Lustre: 181259:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.144@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4139.271719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.709479] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4148.115663] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4152.802388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.808030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4152.812947] Lustre: Skipped 2 previous similar messages [ 4152.813729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4152.817735] Lustre: Skipped 2 previous similar messages [ 4155.612649] Lustre: server umount lustre-MDT0000 complete [ 4157.610521] LustreError: 179423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904790 with bad export cookie 10390814658288995781 [ 4157.617520] LustreError: 179423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4157.617671] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4157.789634] Lustre: server umount lustre-MDT0001 complete [ 4170.430281] Lustre: server umount lustre-OST0000 complete [ 4175.917614] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4181.659355] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4200.392508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4200.533058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4200.554308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4200.602910] Lustre: lustre-MDT0000: new disk, initializing [ 4200.642696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.653898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4202.760994] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4208.676684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.735867] Lustre: 184778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4208.770610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4208.775255] Lustre: Skipped 1 previous similar message [ 4208.815215] Lustre: lustre-MDT0001: new disk, initializing [ 4208.866806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4208.887778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4208.893403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4210.956660] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4216.280709] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4219.665286] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4223.399323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4223.542268] Lustre: lustre-OST0000: new disk, initializing [ 4223.545458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4223.550481] Lustre: 186747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4223.604173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.694070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4225.698505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4225.746514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4226.680889] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4232.138722] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4234.336591] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4234.466960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4236.414705] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4236.515165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4241.379846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.381364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4241.382490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4241.398580] Lustre: Skipped 1 previous similar message [ 4241.402499] LustreError: Skipped 1 previous similar message [ 4243.691448] Lustre: server umount lustre-OST0000 complete [ 4246.497828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.509645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4246.518291] Lustre: Skipped 2 previous similar messages [ 4251.618946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.622956] Lustre: Skipped 1 previous similar message [ 4251.714065] Lustre: server umount lustre-MDT0000 complete [ 4253.591673] LustreError: 184770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904886 with bad export cookie 10390814658288996495 [ 4253.597453] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4253.599374] LustreError: 184770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4253.778241] Lustre: server umount lustre-MDT0001 complete [ 4258.180508] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:01:30 (1788904890) [ 4266.556495] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4267.857453] Key type lgssc unregistered [ 4268.001477] LNet: 189696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4268.004714] LNetError: 189696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4268.016444] LNet: Removed LNI 192.168.204.144@tcp [ 4268.370949] Key type .llcrypt unregistered [ 4268.372778] Key type ._llcrypt unregistered [ 4277.099970] Key type ._llcrypt registered [ 4277.101741] Key type .llcrypt registered [ 4277.152836] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4277.619805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4277.629858] alg: No test for adler32 (adler32-zlib) [ 4278.524714] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 4278.651324] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4280.248173] Key type lgssc registered [ 4280.760989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4284.674388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4284.689843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.957266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.251374] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4292.595210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.790734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4294.904340] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4298.356964] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4300.971737] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4304.178052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4304.260574] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4304.262770] LustreError: 192748:0:(llog.c:1696:llog_backup()) MGC192.168.204.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4304.266350] Lustre: 192748:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.144@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4304.323608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4306.911185] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4309.780158] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4310.716596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.720456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4310.726037] Lustre: Skipped 1 previous similar message [ 4310.731502] Lustre: Skipped 2 previous similar messages [ 4316.130671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4316.135128] Lustre: Skipped 2 previous similar messages [ 4317.153399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4317.157734] Lustre: Skipped 2 previous similar messages [ 4322.273144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4322.276220] Lustre: Skipped 2 previous similar messages [ 4325.344158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4325.430420] Lustre: server umount lustre-MDT0000 complete [ 4326.912543] LustreError: 192775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904960 with bad export cookie 17374319197257266416 [ 4326.917618] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.917775] LustreError: 192775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4327.352273] LustreError: 191616:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005578e2a5 x1875802755978496/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4327.412674] Lustre: server umount lustre-MDT0001 complete [ 4339.359556] Lustre: server umount lustre-OST0000 complete [ 4342.624898] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4346.221257] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4359.570430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.674128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4359.685251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4359.723759] Lustre: lustre-MDT0000: new disk, initializing [ 4359.753756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.765552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4361.412871] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4365.968282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.016797] Lustre: 196270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4366.036640] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4366.038204] Lustre: Skipped 1 previous similar message [ 4366.039896] LustreError: 197052:0:(llog.c:1696:llog_backup()) MGC192.168.204.144@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4366.042594] LustreError: 197052:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4366.045210] Lustre: 197052:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.144@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4366.055685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4366.058469] Lustre: Skipped 1 previous similar message [ 4366.095827] Lustre: lustre-MDT0001: new disk, initializing [ 4366.141118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4366.155971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4366.159495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4367.710883] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4371.702320] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4373.773464] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4376.191371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4376.297579] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4376.299111] Lustre: Skipped 1 previous similar message [ 4376.300888] LustreError: 198212:0:(llog.c:1696:llog_backup()) MGC192.168.204.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4376.305552] LustreError: 198212:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4376.309068] Lustre: 198212:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.144@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4376.313608] Lustre: 198212:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4376.324988] Lustre: lustre-OST0000: new disk, initializing [ 4376.328043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4376.332264] Lustre: 198238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4376.371121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.608737] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4382.475109] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4383.223439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4383.227356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4383.271649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4383.992024] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4384.073714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4385.567885] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4385.653437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4388.321867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4388.324099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.327033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.331182] Lustre: Skipped 2 previous similar messages [ 4392.467106] Lustre: server umount lustre-OST0000 complete [ 4397.024694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4397.026959] LustreError: Skipped 1 previous similar message [ 4397.028166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.032168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.034038] Lustre: Skipped 1 previous similar message [ 4408.288115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4408.358921] Lustre: server umount lustre-MDT0000 complete [ 4408.801288] LustreError: 196276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.807694] LustreError: 196276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4409.687339] LustreError: 196262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905042 with bad export cookie 17374319197257267116 [ 4409.690280] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4409.692062] LustreError: 196262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4409.810800] Lustre: server umount lustre-MDT0001 complete [ 4413.048323] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:04:05 (1788905045) [ 4419.537547] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4423.336777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.463982] LustreError: 201360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.478875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4424.937571] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4427.970321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.623993] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4430.671617] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4433.112130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4435.322850] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4438.376765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.457373] Lustre: lustre-OST0001: new disk, initializing [ 4438.460590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4438.463988] Lustre: 203667:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4440.134830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4440.138838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4440.176712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4440.622480] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4446.233827] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4452.006380] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4452.745235] Lustre: DEBUG MARKER: ADD OST3 [ 4454.531945] loop: module loaded [ 4454.557343] loop0: detected capacity change from 0 to 204800000 [ 4454.685288] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4454.690193] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4454.694116] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4456.782375] loop0: detected capacity change from 0 to 204800000 [ 4458.697938] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4458.762278] Lustre: lustre-OST0003: new disk, initializing [ 4458.764529] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4458.767445] Lustre: 205400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4458.810312] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4458.812838] Lustre: Skipped 3 previous similar messages [ 4459.733726] Lustre: DEBUG MARKER: STOP OST3 [ 4460.366064] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4460.407173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4460.411450] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4460.424961] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4460.463929] Lustre: Failing over lustre-OST0003 [ 4460.509793] Lustre: server umount lustre-OST0003 complete [ 4461.006786] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4462.369057] LustreError: 205744:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4465.031682] Lustre: DEBUG MARKER: START OST3 again [ 4465.136744] LustreError: 202847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.637154] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.638362] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4465.642807] Lustre: Skipped 1 previous similar message [ 4467.360160] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.455490] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4469.415543] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4469.482848] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4469.482934] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4469.493915] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4483.194802] Lustre: Failing over lustre-OST0003 [ 4483.233987] Lustre: server umount lustre-OST0003 complete [ 4485.089254] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4485.089286] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.092718] LustreError: Skipped 1 previous similar message [ 4485.099066] LustreError: 205410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.107332] LustreError: 205410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4493.283679] LustreError: 203022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.290712] LustreError: 203022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4498.405850] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.407656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.411261] Lustre: Skipped 3 previous similar messages [ 4498.416496] Lustre: Skipped 7 previous similar messages [ 4501.928388] Lustre: server umount lustre-MDT0000 complete [ 4503.248577] LustreError: 202494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905136 with bad export cookie 17374319197257267984 [ 4503.253106] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4503.256403] LustreError: 202494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4503.520901] LustreError: 201356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.526107] LustreError: 201356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4509.496797] Lustre: server umount lustre-MDT0001 complete [ 4511.020988] Lustre: server umount lustre-OST0000 complete [ 4512.419975] Lustre: server umount lustre-OST0001 complete [ 4514.766303] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4517.676586] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4530.963511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.056489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4531.068252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4531.106265] Lustre: lustre-MDT0000: new disk, initializing [ 4531.132416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4531.135180] Lustre: Skipped 1 previous similar message [ 4531.141949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4532.646488] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4537.122928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.171117] Lustre: 210808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4537.287178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4538.750877] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4542.669769] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4544.589706] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4546.945497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4547.047562] Lustre: 212776:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4548.285246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4548.288956] Lustre: Skipped 1 previous similar message [ 4548.291235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4548.324467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4549.080369] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4552.893428] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4554.360259] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4554.438561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4555.855141] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4555.927421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4558.818211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4558.819959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4558.820846] LustreError: Skipped 1 previous similar message [ 4558.821345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4558.825765] Lustre: Skipped 4 previous similar messages [ 4558.830269] Lustre: Skipped 5 previous similar messages [ 4562.706481] Lustre: server umount lustre-OST0000 complete [ 4568.032941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4578.272133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4578.331705] Lustre: server umount lustre-MDT0000 complete [ 4579.298064] LustreError: 211610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.304220] LustreError: 211610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4579.759334] LustreError: 210802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905212 with bad export cookie 17374319197257270238 [ 4579.762446] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4579.763869] LustreError: 210802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4579.893643] Lustre: server umount lustre-MDT0001 complete [ 4581.156788] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:06:54 (1788905214) [ 4587.184324] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4590.947892] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4604.261697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.346315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4604.357554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4604.359872] Lustre: Skipped 3 previous similar messages [ 4604.397390] Lustre: lustre-MDT0000: new disk, initializing [ 4604.399673] Lustre: Skipped 2 previous similar messages [ 4604.430110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4605.872735] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4610.257734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.300580] Lustre: 217380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4610.401121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4611.854738] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4615.840144] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4617.867483] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4620.304404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4620.430594] Lustre: 219347:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4621.990036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4622.661383] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4626.612182] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4628.209761] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4628.302720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4629.788539] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4629.872578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4632.547755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.548022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.551547] Lustre: Skipped 2 previous similar messages [ 4632.555419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4632.558519] Lustre: Skipped 6 previous similar messages [ 4636.627121] Lustre: server umount lustre-OST0000 complete [ 4644.056242] Lustre: server umount lustre-MDT0000 complete [ 4645.418264] LustreError: 218175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905278 with bad export cookie 17374319197257271120 [ 4645.421500] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4645.423027] LustreError: 218175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4645.567059] Lustre: server umount lustre-MDT0001 complete [ 4648.929829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.134187] LustreError: 221562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4650.735259] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4653.978991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.719116] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4658.443408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4660.627866] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4663.244043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4663.375768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.378723] Lustre: Skipped 7 previous similar messages [ 4665.616503] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4668.517913] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4686.307590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4686.310034] LustreError: Skipped 2 previous similar messages [ 4691.602692] Lustre: server umount lustre-OST0000 complete [ 4707.296150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4707.367844] Lustre: server umount lustre-MDT0000 complete [ 4708.647321] LustreError: 221544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905341 with bad export cookie 17374319197257272002 [ 4708.650326] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.652824] LustreError: 221544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4708.756445] Lustre: server umount lustre-MDT0001 complete [ 4802.951689] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4803.946782] Key type lgssc unregistered [ 4804.076366] LNet: 225241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4804.078909] LNetError: 225241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4804.087285] LNet: Removed LNI 192.168.204.144@tcp [ 4804.411123] Key type .llcrypt unregistered [ 4804.412517] Key type ._llcrypt unregistered [ 4812.541153] Key type ._llcrypt registered [ 4812.542938] Key type .llcrypt registered [ 4812.581552] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4818.797135] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4819.138091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4819.197580] alg: No test for adler32 (adler32-zlib) [ 4820.062525] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 4820.165900] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4821.752152] Key type lgssc registered [ 4822.111502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4833.445050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4833.450543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.526548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4834.535910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4834.568286] Lustre: lustre-MDT0000: new disk, initializing [ 4834.588728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4834.594770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4835.902724] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4840.027219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.074780] Lustre: 228563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4840.103504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4840.105708] Lustre: Skipped 1 previous similar message [ 4840.140447] Lustre: lustre-MDT0001: new disk, initializing [ 4840.172929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4840.185797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4840.189078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4841.502291] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4845.432072] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4847.361476] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4849.623293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4849.715464] Lustre: lustre-OST0000: new disk, initializing [ 4849.717628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4849.719753] Lustre: 230527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4849.747841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.576946] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4854.268825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4854.272819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4854.303703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4855.316073] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4856.766358] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4856.831455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4858.139790] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4858.212828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4859.364219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4859.365668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4859.366283] LustreError: Skipped 1 previous similar message [ 4859.366612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4859.370844] Lustre: Skipped 1 previous similar message [ 4860.958435] Lustre: server umount lustre-OST0000 complete [ 4864.481696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.482284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4864.485939] Lustre: Skipped 1 previous similar message [ 4864.488583] Lustre: Skipped 2 previous similar messages [ 4868.314407] Lustre: server umount lustre-MDT0000 complete [ 4869.580851] LustreError: 228555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905502 with bad export cookie 14741262722902171563 [ 4869.584545] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4869.586235] LustreError: 228555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4869.703247] Lustre: server umount lustre-MDT0001 complete [ 4875.653102] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4879.046635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.196260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.684047] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4883.531548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.654709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4885.107793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4886.120800] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4888.351870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4890.493225] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4893.439471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4893.505594] Lustre: lustre-OST0001: new disk, initializing [ 4893.507557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4893.510406] Lustre: 235763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4893.544032] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4893.546469] Lustre: Skipped 1 previous similar message [ 4895.513436] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4897.610302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4897.615608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4897.652131] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4901.103211] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4905.804147] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4909.885417] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:12:22 (1788905542) [ 4911.010556] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.012740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4911.014747] Lustre: Skipped 3 previous similar messages [ 4911.016563] Lustre: Skipped 2 previous similar messages [ 4916.193188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4916.195684] Lustre: Skipped 1 previous similar message [ 4916.700753] Lustre: server umount lustre-MDT0000 complete [ 4917.985309] LustreError: 234589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905551 with bad export cookie 14741262722902172445 [ 4917.989172] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4917.989773] LustreError: 234589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4918.110184] Lustre: server umount lustre-MDT0001 complete [ 4929.566844] Lustre: server umount lustre-OST0000 complete [ 4941.460591] Lustre: server umount lustre-OST0001 complete [ 4948.020606] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4959.048980] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5001.299070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.470268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5001.490787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5001.558931] Lustre: lustre-MDT0000: new disk, initializing [ 5001.614216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.627789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5004.417752] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5011.171926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.259125] Lustre: 239711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5011.300597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5011.303550] Lustre: Skipped 1 previous similar message [ 5011.358566] Lustre: lustre-MDT0001: new disk, initializing [ 5011.453279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5011.470559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5014.103077] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5019.676346] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5023.181108] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5026.860222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5027.007371] Lustre: 241679:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5028.251178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5028.263042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5028.321881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5030.459475] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5036.040977] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5038.356454] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5038.464972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5041.087129] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5041.230146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5043.685757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5043.688367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.691217] LustreError: Skipped 1 previous similar message [ 5043.691253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5043.705486] Lustre: Skipped 1 previous similar message [ 5048.489551] Lustre: server umount lustre-OST0000 complete [ 5052.385969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5052.390704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.399489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5052.402921] Lustre: Skipped 1 previous similar message [ 5056.852392] Lustre: server umount lustre-MDT0000 complete [ 5058.966462] LustreError: 239704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905692 with bad export cookie 14741262722902174566 [ 5058.969617] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5058.976376] LustreError: 239704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5059.161731] Lustre: server umount lustre-MDT0001 complete [ 5064.262451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5064.550186] LustreError: 243892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.562836] LustreError: 243892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5064.594922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.600777] Lustre: Skipped 2 previous similar messages [ 5067.022170] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5069.794309] LustreError: 243893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5071.566908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.270137] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5078.419259] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5081.270061] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5084.613293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5088.001889] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5092.034964] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5237.731265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5237.740738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.749524] Lustre: Skipped 1 previous similar message [ 5237.753280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5237.761933] Lustre: Skipped 2 previous similar messages [ 5241.628731] Lustre: server umount lustre-OST0000 complete [ 5254.977218] Lustre: server umount lustre-MDT0000 complete [ 5255.145120] LustreError: 243893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.169330] LustreError: 243893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5259.810466] LustreError: 243874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905892 with bad export cookie 14741262722902175448 [ 5259.835222] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5259.840380] LustreError: 243874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5260.611504] Lustre: server umount lustre-MDT0001 complete [ 5360.469033] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5363.491290] Key type lgssc unregistered [ 5363.787314] LNet: 247493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5363.798739] LNetError: 247493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5363.819844] LNet: Removed LNI 192.168.204.144@tcp [ 5364.811155] Key type .llcrypt unregistered [ 5364.813162] Key type ._llcrypt unregistered [ 5389.574215] Key type ._llcrypt registered [ 5389.579316] Key type .llcrypt registered [ 5389.678605] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5405.947992] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5406.894716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5406.914604] alg: No test for adler32 (adler32-zlib) [ 5408.173641] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 5408.462460] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5410.136375] Key type lgssc registered [ 5411.248974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5460.900478] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5472.021353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5472.047689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.359344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5473.408500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5473.495581] Lustre: lustre-MDT0000: new disk, initializing [ 5473.599932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5473.631431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5478.640648] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5489.883653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5490.011201] Lustre: 251564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5490.075949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5490.080291] Lustre: Skipped 1 previous similar message [ 5490.186442] Lustre: lustre-MDT0001: new disk, initializing [ 5490.303129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5490.338710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5490.353045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5496.040543] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5501.429900] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5509.528359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5509.932643] Lustre: lustre-OST0000: new disk, initializing [ 5509.943170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5509.950714] Lustre: 253196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5510.096116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.615775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5511.643256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5511.789676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5517.481537] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5530.130095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5530.342454] Lustre: lustre-OST0001: new disk, initializing [ 5530.351461] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5530.368362] Lustre: 254067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5530.506392] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5536.442780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5536.469226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5536.628939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5538.341964] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5551.831829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5557.897228] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5572.068150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5572.074604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.085300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5572.591932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.595629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5572.610072] Lustre: Skipped 1 previous similar message [ 5576.337214] Lustre: server umount lustre-MDT0000 complete [ 5580.573102] LustreError: 251555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906213 with bad export cookie 8719789509636580992 [ 5580.575325] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5580.594235] LustreError: 251555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5581.051666] Lustre: server umount lustre-MDT0001 complete [ 5595.970913] Lustre: server umount lustre-OST0000 complete [ 5598.176301] Lustre: 249069:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788906215/real 1788906215] req@ffff8959e6867100 x1875803940929536/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788906231 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5598.197641] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.209531] Lustre: Skipped 1 previous similar message [ 5600.077964] Lustre: server umount lustre-OST0001 complete [ 5614.081922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5623.976947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5632.301287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5642.188926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5663.660788] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5674.493426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.518975] Lustre: lustre-MDT0000: reset Object Index mappings [ 5674.970780] LustreError: 259078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5674.987124] LustreError: 259078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5675.054591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.586518] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5680.103708] LustreError: 259079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.122501] LustreError: 259079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5685.224626] LustreError: 259078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5689.110922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.147481] Lustre: lustre-MDT0001: reset Object Index mappings [ 5689.702170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.282727] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5698.689173] Lustre: 260220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5705.967727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5706.329879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.886470] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5714.541599] LustreError: 260576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.049747] LustreError: 261018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.086633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5722.267143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5727.747513] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5730.951951] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5739.140787] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5742.942242] Lustre: 262094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5748.872465] Lustre: server umount lustre-MDT0000 complete [ 5751.269051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5751.274262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5751.286147] LustreError: 259078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.304570] LustreError: 259078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5753.137568] LustreError: 262096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906386 with bad export cookie 8719789509636587782 [ 5753.138169] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5753.154714] LustreError: 262096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5753.323747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5753.329757] Lustre: Skipped 3 previous similar messages [ 5753.569560] Lustre: server umount lustre-MDT0001 complete [ 5759.885354] Lustre: server umount lustre-OST0000 complete [ 5763.518752] Lustre: server umount lustre-OST0001 complete [ 5782.535467] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:26:54 (1788906414) [ 5803.773662] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5819.143544] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5866.196472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.442670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5866.488391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5866.608747] Lustre: lustre-MDT0000: new disk, initializing [ 5866.691221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5866.700195] Lustre: Skipped 1 previous similar message [ 5866.711698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5872.027990] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5883.700487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.898727] Lustre: 266366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5883.975618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5883.983078] Lustre: Skipped 1 previous similar message [ 5884.089890] Lustre: lustre-MDT0001: new disk, initializing [ 5884.332236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5884.350093] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5888.970929] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5900.485191] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5906.833693] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5914.514828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5914.819612] Lustre: lustre-OST0000: new disk, initializing [ 5914.824562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5914.863387] Lustre: 268333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5916.210288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5916.238515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5916.356980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5921.700593] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5932.840104] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5938.103883] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5938.377771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5944.042397] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5944.266607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5947.361928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5947.361988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5947.374048] LustreError: Skipped 1 previous similar message [ 5947.385736] Lustre: Skipped 4 previous similar messages [ 5947.388589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5947.416620] Lustre: Skipped 1 previous similar message [ 5952.494929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5952.508263] Lustre: Skipped 1 previous similar message [ 5952.755559] Lustre: server umount lustre-OST0000 complete [ 5957.607929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.624288] Lustre: Skipped 2 previous similar messages [ 5962.728174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5962.736050] Lustre: Skipped 3 previous similar messages [ 5963.547782] Lustre: server umount lustre-MDT0000 complete [ 5967.377849] LustreError: 266358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906600 with bad export cookie 8719789509636589875 [ 5967.384414] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5967.396584] LustreError: 266358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5967.828434] Lustre: server umount lustre-MDT0001 complete [ 5989.485951] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5999.754519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.107449] LustreError: 271269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.124445] LustreError: 271269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6000.183406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6000.196134] Lustre: Skipped 2 previous similar messages [ 6004.686978] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6014.520257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.781382] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6021.527684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6027.675104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6033.827373] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6041.259895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6041.399978] Lustre: lustre-OST0001: new disk, initializing [ 6041.405285] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6041.413219] Lustre: 273577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6042.641711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6042.649663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6042.753516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6046.685113] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6056.234583] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6059.914981] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6064.519450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6064.633116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6076.385532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6076.394200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.413168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6079.910899] Lustre: server umount lustre-MDT0000 complete [ 6082.026941] LustreError: 271269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.058679] LustreError: 271269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6084.078592] LustreError: 271249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906717 with bad export cookie 8719789509636590757 [ 6084.091232] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6084.092195] LustreError: 271249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6084.668393] Lustre: server umount lustre-MDT0001 complete [ 6098.062248] Lustre: server umount lustre-OST0000 complete [ 6112.974824] Lustre: server umount lustre-OST0001 complete [ 6129.896533] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6140.001897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.301317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6140.307620] Lustre: Skipped 3 previous similar messages [ 6144.722339] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6150.630176] LustreError: 276924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.646207] LustreError: 276924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6154.165452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.935451] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6161.917042] Lustre: 278064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6168.233415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.710816] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6181.121728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6185.514281] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6185.515413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6187.173475] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6194.310501] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6198.689691] Lustre: 279944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6204.794939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6204.862581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6220.373868] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:34:11 (1788906851) [ 6236.649335] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.652733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6236.653426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6236.667375] Lustre: Skipped 6 previous similar messages [ 6236.698924] Lustre: Skipped 6 previous similar messages [ 6242.416304] Lustre: server umount lustre-MDT0000 complete [ 6246.208188] LustreError: 277666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906879 with bad export cookie 8719789509636597141 [ 6246.220432] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6246.224974] LustreError: 277666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6252.692826] Lustre: server umount lustre-MDT0001 complete [ 6262.746838] Lustre: server umount lustre-OST0000 complete [ 6266.628351] Lustre: server umount lustre-OST0001 complete [ 6273.540870] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 6284.390510] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6323.949990] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6332.487168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6332.682731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6332.714913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6332.787569] Lustre: lustre-MDT0000: new disk, initializing [ 6332.873966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6336.889956] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6346.366383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.456863] Lustre: 284468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6346.643087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6350.323828] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6359.896366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6360.124391] Lustre: 286101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6362.107595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6365.189532] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6374.332303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6374.495572] Lustre: 286974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6376.124994] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6379.856408] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6388.244388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6390.874902] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6390.879774] Lustre: Skipped 1 previous similar message [ 6400.726205] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:37:13 (1788907033) [ 6402.020335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.022884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.027334] Lustre: Skipped 3 previous similar messages [ 6402.040939] Lustre: Skipped 11 previous similar messages [ 6408.237591] Lustre: server umount lustre-MDT0000 complete [ 6411.078073] LustreError: 284462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907044 with bad export cookie 8719789509636605065 [ 6411.079110] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.085950] LustreError: 284462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6411.333155] Lustre: server umount lustre-MDT0001 complete [ 6424.431853] Lustre: server umount lustre-OST0000 complete [ 6437.866827] Lustre: server umount lustre-OST0001 complete [ 6444.732788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6445.044309] LustreError: 289219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.069893] LustreError: 289219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6445.153069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.159261] Lustre: Skipped 7 previous similar messages [ 6448.274789] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6454.865220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.375690] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6463.542370] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6467.507963] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6472.574320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6477.060642] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6482.352974] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6506.292435] loop1: detected capacity change from 0 to 204800000 [ 6506.498454] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6506.506101] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6506.522562] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6511.088473] loop1: detected capacity change from 0 to 204800000 [ 6516.118075] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.196314] LustreError: 289208:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs3082: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6516.215073] LustreError: 289208:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs3082 [ 6516.219350] LustreError: 289208:0:(mgs_handler.c:517:mgs_target_reg()) fs3082-MDT0001: Error checking target index: rc = -13 [ 6516.226606] LustreError: fs3082-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6516.232877] LustreError: 292955:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 6516.238902] LustreError: 292955:0:(tgt_mount.c:2132:server_put_super()) no obd fs3082-MDT0001 [ 6516.243170] LustreError: 292955:0:(tgt_mount.c:116:server_deregister_mount()) fs3082-MDT0001 not registered [ 6516.287600] Lustre: server umount fs3082-MDT0001 complete [ 6516.292325] LustreError: 292955:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6522.786974] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.871714] Lustre: Setting parameter fs3082-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3082-MDT0001 [ 6522.927533] Lustre: srv-fs3082-MDT0001: No data found on store. Initialize space. [ 6522.931269] Lustre: Skipped 4 previous similar messages [ 6523.035655] Lustre: fs3082-MDT0001: new disk, initializing [ 6523.043893] Lustre: Skipped 3 previous similar messages [ 6523.154410] Lustre: srv-fs3082-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6523.168034] Lustre: 293422:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3082-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6523.174168] LustreError: 293422:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3082-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6527.044400] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6533.300283] Lustre: server umount fs3082-MDT0001 complete [ 6544.490393] loop1: detected capacity change from 0 to 204800000 [ 6549.251764] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.559781] Lustre: 294988:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3082-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6549.567081] LustreError: 294988:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3082-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6553.299667] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6557.139034] Lustre: server umount fs3082-MDT0001 complete [ 6563.303892] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6567.907988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6567.922079] Lustre: Skipped 2 previous similar messages [ 6567.928039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6567.935031] Lustre: Skipped 4 previous similar messages [ 6571.517633] Lustre: server umount lustre-MDT0000 complete [ 6578.466470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6578.586900] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6582.722871] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6584.318824] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6588.652309] loop1: detected capacity change from 0 to 204800000 [ 6593.225892] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.498503] Lustre: 297430:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3082-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6593.509710] LustreError: 297430:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3082-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6597.057218] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6601.246648] Lustre: server umount fs3082-MDT0001 complete [ 6618.594371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6618.600049] LustreError: Skipped 1 previous similar message [ 6631.392413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6631.513777] Lustre: server umount lustre-OST0000 complete [ 6649.312866] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6650.059605] Lustre: server umount lustre-MDT0000 complete [ 6655.080682] LustreError: 289201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907288 with bad export cookie 8719789509636607746 [ 6655.085930] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6655.093925] LustreError: 289201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6655.912449] Lustre: server umount lustre-MDT0001 complete [ 6669.988861] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6674.004571] Key type lgssc unregistered [ 6674.518571] LNet: 299775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6674.529379] LNetError: 299775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6674.556136] LNet: Removed LNI 192.168.204.144@tcp [ 6675.764191] Key type .llcrypt unregistered [ 6675.767585] Key type ._llcrypt unregistered [ 6704.445812] Key type ._llcrypt registered [ 6704.448788] Key type .llcrypt registered [ 6704.605815] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6706.384791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6706.438968] alg: No test for adler32 (adler32-zlib) [ 6707.632340] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 6707.908903] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6709.608231] Key type lgssc registered [ 6710.901343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6719.870974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6719.896279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6721.704738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.636591] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6731.234179] LustreError: 301044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.245701] LustreError: 301044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6735.793113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.157240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.353566] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6746.779540] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6752.164587] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6758.227871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6758.644848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6761.706636] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.609474] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6766.824376] LustreError: 302875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6766.844778] LustreError: 302875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6771.393781] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6772.202630] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6772.215298] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6777.320435] LustreError: 302876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.338886] LustreError: 302876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6787.556977] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6787.574918] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6788.347686] loop1: detected capacity change from 0 to 204800000 [ 6788.517473] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6788.536135] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6788.558185] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6793.668129] loop1: detected capacity change from 0 to 204800000 [ 6799.677096] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6799.797496] Lustre: fs17018-OST0000: new disk, initializing [ 6799.812946] Lustre: srv-fs17018-OST0000: No data found on store. Initialize space. [ 6799.822929] Lustre: 304780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6799.904852] Lustre: fs17018-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6804.974273] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6804.988566] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6806.791379] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6815.456454] Lustre: server umount fs17018-OST0000 complete [ 6829.223026] loop1: detected capacity change from 0 to 204800000 [ 6835.457621] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6835.762887] Lustre: fs17018-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6838.773909] LustreError: 302876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.794537] LustreError: 302876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 6842.339918] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6848.997171] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 6851.889772] loop2: detected capacity change from 0 to 204800000 [ 6852.130124] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6852.143334] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6852.159273] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6857.642384] loop2: detected capacity change from 0 to 204800000 [ 6863.225323] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6863.270548] LustreError: fs17018-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6863.281288] LustreError: fs17018-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6863.288374] LustreError: 307824:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 6863.299593] LustreError: 307824:0:(tgt_mount.c:2132:server_put_super()) no obd fs17018-OST0001 [ 6863.308875] LustreError: 307824:0:(tgt_mount.c:116:server_deregister_mount()) fs17018-OST0001 not registered [ 6863.345992] Lustre: server umount fs17018-OST0001 complete [ 6863.349661] LustreError: 307824:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6873.382585] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.621580] Lustre: fs17018-OST0001: new disk, initializing [ 6873.631950] Lustre: srv-fs17018-OST0001: No data found on store. Initialize space. [ 6873.640028] Lustre: 308375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6873.743532] Lustre: fs17018-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6879.806613] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6888.727580] Lustre: server umount fs17018-OST0000 complete [ 6897.120924] Lustre: server umount fs17018-OST0001 complete [ 6905.317424] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6905.334017] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 6909.446349] loop1: detected capacity change from 0 to 204800000 [ 6915.069311] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6915.329421] Lustre: fs17018-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.215683] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6928.282277] loop2: detected capacity change from 0 to 204800000 [ 6934.341220] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6940.249757] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6948.874768] Lustre: server umount fs17018-OST0000 complete [ 6958.499378] Lustre: server umount fs17018-OST0001 complete [ 6971.892545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.900646] LustreError: Skipped 1 previous similar message [ 6971.902592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.905316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6971.925240] Lustre: Skipped 1 previous similar message [ 6977.015544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6977.036391] Lustre: Skipped 2 previous similar messages [ 6977.810307] Lustre: server umount lustre-OST0000 complete [ 6982.114175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.126737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.126989] Lustre: Skipped 1 previous similar message [ 6982.142367] Lustre: Skipped 1 previous similar message [ 6987.234837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6987.243300] Lustre: Skipped 1 previous similar message [ 6990.161181] Lustre: server umount lustre-MDT0000 complete [ 6994.131736] LustreError: 301786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907627 with bad export cookie 48544233286207351 [ 6994.139968] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6994.143397] LustreError: 301786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6994.467698] Lustre: server umount lustre-MDT0001 complete [ 7005.109789] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7007.857164] Key type lgssc unregistered [ 7008.232063] LNet: 314294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7008.240693] LNetError: 314294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7009.323418] LNet: Removed LNI 192.168.204.144@tcp [ 7010.360157] Key type .llcrypt unregistered [ 7010.364716] Key type ._llcrypt unregistered [ 7039.435988] Key type ._llcrypt registered [ 7039.442910] Key type .llcrypt registered [ 7039.537517] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7053.051399] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7053.789791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7053.850494] alg: No test for adler32 (adler32-zlib) [ 7054.838763] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 7055.054291] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7056.696171] Key type lgssc registered [ 7057.571700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7083.871693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7083.885197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.036284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7085.058507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7085.108991] Lustre: lustre-MDT0000: new disk, initializing [ 7085.166207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.182169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7088.359334] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7096.259309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.341383] Lustre: 318149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7096.396916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7096.399977] Lustre: Skipped 1 previous similar message [ 7096.469635] Lustre: lustre-MDT0001: new disk, initializing [ 7096.528334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7096.559890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7096.570754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7099.147790] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7105.820419] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7109.673980] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7113.908861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7114.089328] Lustre: lustre-OST0000: new disk, initializing [ 7114.092829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7114.098826] Lustre: 320117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7114.165171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7118.165186] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7122.991461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7122.996744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7123.079983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7124.386723] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7125.850960] Lustre: Mounted lustre-client - version 2.17.58_5_gca60a6d [ 7126.542662] Lustre: Unmounted lustre-client [ 7133.153511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7133.160624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7133.175567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7133.665755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7133.671166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7133.684105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7136.228783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7136.233075] Lustre: Skipped 1 previous similar message [ 7138.573613] Lustre: server umount lustre-OST0000 complete [ 7141.346249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.353442] Lustre: Skipped 1 previous similar message [ 7141.354818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.360958] Lustre: Skipped 1 previous similar message [ 7146.466872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7146.470590] Lustre: Skipped 1 previous similar message [ 7155.168155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7155.273173] Lustre: server umount lustre-MDT0000 complete [ 7157.456939] LustreError: 318139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907790 with bad export cookie 13237879715757486633 [ 7157.459334] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7157.463726] LustreError: 318139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7157.646556] Lustre: server umount lustre-MDT0001 complete [ 7164.688103] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7166.667763] Key type lgssc unregistered [ 7185.313570] LNet: 322279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7185.321355] LNetError: 322279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7255.976486] LNet: Removed LNI 192.168.204.144@tcp [ 7256.709842] Key type .llcrypt unregistered [ 7256.713718] Key type ._llcrypt unregistered [ 7269.831637] Key type ._llcrypt registered [ 7269.833539] Key type .llcrypt registered [ 7269.921887] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7282.174438] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7282.838730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7282.923114] alg: No test for adler32 (adler32-zlib) [ 7283.936888] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 7284.190727] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7285.880223] Key type lgssc registered [ 7286.754389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7309.546136] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:52:21 (1788907941) [ 7314.289340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7314.317244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.457928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7315.482165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7315.535835] Lustre: lustre-MDT0000: new disk, initializing [ 7315.601426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.619900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7318.633646] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7325.897236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.971568] Lustre: 325473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7326.008915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7326.015885] Lustre: Skipped 1 previous similar message [ 7326.066724] Lustre: lustre-MDT0001: new disk, initializing [ 7326.122370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7326.147032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7326.154950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7328.731903] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7334.723537] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7338.237407] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7342.652956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7342.835979] Lustre: lustre-OST0000: new disk, initializing [ 7342.840773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7342.848404] Lustre: 327438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7342.904806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.701452] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7350.297209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7350.311051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7350.392126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7352.983987] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7365.610165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7365.616155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7365.620531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7365.636532] Lustre: Skipped 1 previous similar message [ 7368.780210] Lustre: server umount lustre-OST0000 complete [ 7372.259745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7372.267877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.274561] Lustre: Skipped 1 previous similar message [ 7372.278221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7375.842261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7375.844345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.528748] Lustre: server umount lustre-MDT0000 complete [ 7379.872066] LustreError: 327437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908012 with bad export cookie 11035299517766115498 [ 7379.876587] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7379.878732] LustreError: 327437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7380.092914] Lustre: server umount lustre-MDT0001 complete [ 7386.574380] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7388.203536] Key type lgssc unregistered [ 7388.413706] LNet: 329343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7388.422983] LNetError: 329343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7388.435144] LNet: Removed LNI 192.168.204.144@tcp [ 7388.955177] Key type .llcrypt unregistered [ 7388.957959] Key type ._llcrypt unregistered [ 7402.609791] Key type ._llcrypt registered [ 7402.611888] Key type .llcrypt registered [ 7402.712719] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7403.548976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7403.576394] alg: No test for adler32 (adler32-zlib) [ 7404.599565] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 7404.788344] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7406.432192] Key type lgssc registered [ 7407.283684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7413.035396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7413.051809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.318188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.527396] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7423.308188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.600436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7426.993965] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7432.075322] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7435.583660] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7439.649265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7439.891797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7443.346852] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7447.808304] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7465.731603] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7466.793312] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7469.538385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7469.545921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7469.546720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7469.549242] Lustre: Skipped 2 previous similar messages [ 7470.082321] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 7470.092230] Lustre: Skipped 2 previous similar messages [ 7474.658937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7474.665033] Lustre: Skipped 2 previous similar messages [ 7475.567716] Lustre: server umount lustre-MDT0000 complete [ 7478.286093] LustreError: 330572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908111 with bad export cookie 16294258647329948514 [ 7478.297852] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7478.301050] LustreError: 330572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7479.779073] LustreError: 332891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7479.783610] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.799438] LustreError: 332891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7479.814697] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7479.821867] Lustre: Skipped 1 previous similar message [ 7480.294058] LustreError: 330588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.809580] LustreError: 332891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7484.698449] Lustre: server umount lustre-MDT0001 complete [ 7494.000712] Lustre: server umount lustre-OST0000 complete [ 7500.624717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7503.911840] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7506.152785] Lustre: 334792:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.144@tcp [ 7507.710358] Lustre: server umount MGS complete [ 7513.672414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.940755] LustreError: 335364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7513.995499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7517.019451] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7519.202856] LustreError: 335365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7522.636629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.909551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7525.717543] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7530.759430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7530.984049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.116594] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7539.855663] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7540.785994] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7544.971794] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7546.114669] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7549.413262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7549.418078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7549.423853] Lustre: Skipped 1 previous similar message [ 7549.428392] Lustre: Skipped 2 previous similar messages [ 7555.220819] Lustre: server umount lustre-OST0000 complete [ 7558.626293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7558.635504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7558.650138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7558.654224] Lustre: Skipped 2 previous similar messages [ 7563.843408] Lustre: server umount lustre-MDT0000 complete [ 7564.772510] LustreError: 335364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.788612] LustreError: 335364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7566.387414] LustreError: 335345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908199 with bad export cookie 16294258647329949347 [ 7566.392160] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7566.395178] LustreError: 335345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7566.635969] Lustre: server umount lustre-MDT0001 complete [ 7574.611492] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7576.549085] Key type lgssc unregistered [ 7576.754701] LNet: 338795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7576.759253] LNetError: 338795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7576.770379] LNet: Removed LNI 192.168.204.144@tcp [ 7577.391224] Key type .llcrypt unregistered [ 7577.393181] Key type ._llcrypt unregistered [ 7588.768110] Key type ._llcrypt registered [ 7588.769825] Key type .llcrypt registered [ 7588.827628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7598.225103] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7598.935588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7598.973964] alg: No test for adler32 (adler32-zlib) [ 7599.932138] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 7600.120063] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7601.785172] Key type lgssc registered [ 7602.705365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7627.476816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7627.493645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.664537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7628.689756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7628.744213] Lustre: lustre-MDT0000: new disk, initializing [ 7628.798066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.809639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7631.671401] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7639.534877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7639.631157] Lustre: 341936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7639.673546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7639.678141] Lustre: Skipped 1 previous similar message [ 7639.758701] Lustre: lustre-MDT0001: new disk, initializing [ 7639.867856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7639.907907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7639.913922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7643.336515] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7651.053409] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7655.679585] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7660.711365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7660.959640] Lustre: lustre-OST0000: new disk, initializing [ 7660.964198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7660.970577] Lustre: 343905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7661.067840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7665.650244] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7669.297615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7669.307102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7669.395287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7672.825789] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7676.619840] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7676.770794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7680.159843] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7680.332967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7684.578505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7684.585235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7684.599273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7685.095246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7685.102303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7685.117716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7685.120373] Lustre: Skipped 1 previous similar message [ 7687.836151] Lustre: server umount lustre-OST0000 complete [ 7691.237691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7691.246228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.255848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7695.329724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7695.331330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.625301] Lustre: server umount lustre-MDT0000 complete [ 7699.026604] LustreError: 341928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908332 with bad export cookie 141945350859810795 [ 7699.033521] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7699.035054] LustreError: 341928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7699.234708] Lustre: server umount lustre-MDT0001 complete [ 7704.792575] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:58:57 (1788908337) [ 7714.826690] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7720.692689] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7743.253130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.404407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7743.429583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7743.478597] Lustre: lustre-MDT0000: new disk, initializing [ 7743.521795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.566703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7746.156466] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7752.942868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.021764] Lustre: 348660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7753.072431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7753.075146] Lustre: Skipped 1 previous similar message [ 7753.088104] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7753.091823] Lustre: Skipped 1 previous similar message [ 7753.134904] Lustre: lustre-MDT0001: new disk, initializing [ 7753.210817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7753.248898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7753.255206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7756.121829] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7762.423224] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7766.124371] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7770.352779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7770.501981] Lustre: lustre-OST0000: new disk, initializing [ 7770.505506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7770.513429] Lustre: 350626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7770.567549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7773.579237] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7779.000657] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7780.369481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7780.374630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7780.437383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7800.805551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7800.806524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.811543] LustreError: Skipped 1 previous similar message [ 7800.821829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.746911] Lustre: server umount lustre-OST0000 complete [ 7807.288860] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7811.042036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7811.053089] Lustre: Skipped 2 previous similar messages [ 7813.528890] Lustre: server umount lustre-MDT0000 complete [ 7815.747900] LustreError: 351493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908448 with bad export cookie 141945350859811677 [ 7815.751906] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7815.754558] LustreError: 351493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7815.954113] Lustre: server umount lustre-MDT0001 complete [ 7821.599987] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7823.185471] Key type lgssc unregistered [ 7823.367616] LNet: 352774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7823.374473] LNetError: 352774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7823.385909] LNet: Removed LNI 192.168.204.144@tcp [ 7823.874054] Key type .llcrypt unregistered [ 7823.876171] Key type ._llcrypt unregistered [ 7875.601936] Key type ._llcrypt registered [ 7875.604360] Key type .llcrypt registered [ 7875.663337] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7875.825583] Key type .llcrypt unregistered [ 7875.827454] Key type ._llcrypt unregistered [ 7921.249606] Key type ._llcrypt registered [ 7921.251773] Key type .llcrypt registered [ 7921.329337] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7921.517299] Key type .llcrypt unregistered [ 7921.518988] Key type ._llcrypt unregistered [ 7960.095183] Key type ._llcrypt registered [ 7960.105372] Key type .llcrypt registered [ 7960.348658] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7960.823332] Key type .llcrypt unregistered [ 7960.825954] Key type ._llcrypt unregistered [ 8002.709402] Key type ._llcrypt registered [ 8002.713793] Key type .llcrypt registered [ 8002.810488] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8003.163969] Key type .llcrypt unregistered [ 8003.168933] Key type ._llcrypt unregistered [ 8082.297853] Key type ._llcrypt registered [ 8082.300371] Key type .llcrypt registered [ 8082.413027] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8082.708865] Key type .llcrypt unregistered [ 8082.714841] Key type ._llcrypt unregistered [ 8114.275222] Key type ._llcrypt registered [ 8114.277729] Key type .llcrypt registered [ 8114.392102] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8114.645804] Key type .llcrypt unregistered [ 8114.648446] Key type ._llcrypt unregistered [ 8145.675207] Key type ._llcrypt registered [ 8145.677712] Key type .llcrypt registered [ 8145.818979] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8146.057307] Key type .llcrypt unregistered [ 8146.059401] Key type ._llcrypt unregistered [ 8176.657903] Key type ._llcrypt registered [ 8176.660177] Key type .llcrypt registered [ 8176.763484] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8177.098297] Key type .llcrypt unregistered [ 8177.100772] Key type ._llcrypt unregistered [ 8208.421572] Key type ._llcrypt registered [ 8208.426937] Key type .llcrypt registered [ 8208.529166] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8208.813515] Key type .llcrypt unregistered [ 8208.815897] Key type ._llcrypt unregistered [ 8240.659628] Key type ._llcrypt registered [ 8240.661805] Key type .llcrypt registered [ 8240.767538] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8241.089494] Key type .llcrypt unregistered [ 8241.101502] Key type ._llcrypt unregistered [ 8302.841090] Key type ._llcrypt registered [ 8302.842867] Key type .llcrypt registered [ 8303.016849] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8303.365465] Key type .llcrypt unregistered [ 8303.367408] Key type ._llcrypt unregistered [ 8338.744656] Key type ._llcrypt registered [ 8338.751562] Key type .llcrypt registered [ 8338.859466] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8339.138564] Key type .llcrypt unregistered [ 8339.143960] Key type ._llcrypt unregistered [ 8370.158448] Key type ._llcrypt registered [ 8370.163231] Key type .llcrypt registered [ 8370.277723] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8370.662465] Key type .llcrypt unregistered [ 8370.666483] Key type ._llcrypt unregistered [ 8423.045758] Key type ._llcrypt registered [ 8423.055749] Key type .llcrypt registered [ 8423.156292] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8440.893673] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8441.874707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8441.895889] alg: No test for adler32 (adler32-zlib) [ 8442.924896] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 8443.108109] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8444.752146] Key type lgssc registered [ 8445.665937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8479.080859] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8479.108439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8480.341626] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8480.363526] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8480.417742] Lustre: tst250fs-MDT0000: new disk, initializing [ 8480.476804] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.495649] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8484.918640] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8494.232296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.341631] Lustre: 372993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8494.405403] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8494.408943] Lustre: Skipped 1 previous similar message [ 8494.489643] Lustre: tst250fs-MDT0001: new disk, initializing [ 8494.574508] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8494.613256] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8494.627210] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8498.644028] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8503.268860] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8509.235321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8509.540036] Lustre: tst250fs-OST0000: new disk, initializing [ 8509.545188] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8509.555024] Lustre: 374625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8509.566419] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [ 8509.656885] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.077262] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8515.089829] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8515.167881] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8515.280411] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8524.734795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8524.894868] Lustre: tst250fs-OST0001: new disk, initializing [ 8524.914945] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8524.929142] Lustre: 375496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8525.055884] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8525.950861] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8525.957731] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8526.103443] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8530.955281] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8536.084663] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8538.914573] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [ 8538.922233] Lustre: Skipped 2 previous similar messages [ 8556.003473] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8556.010583] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.019970] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8557.031527] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8557.034186] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8557.047920] Lustre: Skipped 1 previous similar message [ 8562.151187] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8562.159854] Lustre: Skipped 5 previous similar messages [ 8567.268335] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8567.278678] Lustre: Skipped 4 previous similar messages [ 8569.312165] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8571.536710] Lustre: server umount tst250fs-MDT0000 complete [ 8574.659470] LustreError: 372986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909207 with bad export cookie 5926113297773796896 [ 8574.660532] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8574.672920] LustreError: 372986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8574.962460] Lustre: server umount tst250fs-MDT0001 complete [ 8588.201359] Lustre: server umount tst250fs-OST0000 complete [ 8601.655387] Lustre: server umount tst250fs-OST0001 complete [ 8616.566477] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8623.540569] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8649.073834] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:14:40 (1788909280) [ 8650.626189] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8666.149501] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8676.475787] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8705.248075] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8579 sec ========== 19:15:37 (1788909337) [ 8706.689517] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:15:38 (1788909338) === [ 8709.648544] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:15:41 (1788909341) === [ 8739.513501] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8741.581667] Key type lgssc unregistered [ 8741.809459] LNet: 384624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8741.814816] LNetError: 384624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8741.830910] LNet: Removed LNI 192.168.204.144@tcp [ 8742.482261] Key type .llcrypt unregistered [ 8742.485333] Key type ._llcrypt unregistered