[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 463323307 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002273] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007011] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008005] pid_max: default: 32768 minimum: 301 [ 0.009092] LSM: Security Framework initializing [ 0.010029] Yama: becoming mindful. [ 0.011021] SELinux: Initializing. [ 0.012055] *** VALIDATE selinux *** [ 0.020144] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025418] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027075] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028081] *** VALIDATE tmpfs *** [ 0.029406] *** VALIDATE proc *** [ 0.030149] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.032212] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035017] Spectre V2 : User space: Vulnerable [ 0.036004] Speculative Store Bypass: Vulnerable [ 0.039475] debug: unmapping init [mem 0xffffffffa7259000-0xffffffffa7260fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042670] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043011] ... version: 2 [ 0.043939] ... bit width: 48 [ 0.044010] ... generic registers: 4 [ 0.045008] ... value mask: 0000ffffffffffff [ 0.046007] ... max period: 00007fffffffffff [ 0.047006] ... fixed-purpose events: 3 [ 0.047860] ... event mask: 000000070000000f [ 0.048249] rcu: Hierarchical SRCU implementation. [ 0.050542] smp: Bringing up secondary CPUs ... [ 0.051567] x86: Booting SMP configuration: [ 0.052011] .... node #0, CPUs: #1 #2 #3 [ 0.054715] smp: Brought up 1 node, 4 CPUs [ 0.055853] smpboot: Max logical packages: 1 [ 0.056006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153501] node 0 deferred pages initialised in 96ms [ 0.155615] devtmpfs: initialized [ 0.156202] x86/mm: Memory block size: 128MB [ 0.158244] gcov: version magic: 0x41383552 [ 0.159547] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160053] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161149] pinctrl core: initialized pinctrl subsystem [ 0.162097] [ 0.162412] ************************************************************* [ 0.163007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164006] ** ** [ 0.165006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166006] ** ** [ 0.167005] ** This means that this kernel is built to expose internal ** [ 0.168005] ** IOMMU data structures, which may compromise security on ** [ 0.169006] ** your system. ** [ 0.170006] ** ** [ 0.171008] ** If you see this message and you are not debugging the ** [ 0.172007] ** kernel, report this immediately to your vendor! ** [ 0.173006] ** ** [ 0.174007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175006] ************************************************************* [ 0.176580] NET: Registered protocol family 16 [ 0.177451] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.178042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180429] cpuidle: using governor menu [ 0.182465] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185459] PCI: Using configuration type 1 for base access [ 0.187110] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196086] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201096] cryptd: max_cpu_qlen set to 1000 [ 0.202164] ACPI: Added _OSI(Module Device) [ 0.203008] ACPI: Added _OSI(Processor Device) [ 0.203892] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210326] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216685] ACPI: Interpreter enabled [ 0.218051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220009] ACPI: Using IOAPIC for interrupt routing [ 0.222092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.238329] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244032] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246893] acpiphp: Slot [2] registered [ 0.248071] acpiphp: Slot [5] registered [ 0.248901] acpiphp: Slot [6] registered [ 0.250049] acpiphp: Slot [7] registered [ 0.250842] acpiphp: Slot [8] registered [ 0.251047] acpiphp: Slot [9] registered [ 0.251841] acpiphp: Slot [10] registered [ 0.253048] acpiphp: Slot [3] registered [ 0.253818] acpiphp: Slot [4] registered [ 0.254058] acpiphp: Slot [11] registered [ 0.254867] acpiphp: Slot [12] registered [ 0.256043] acpiphp: Slot [13] registered [ 0.257046] acpiphp: Slot [14] registered [ 0.258055] acpiphp: Slot [15] registered [ 0.259051] acpiphp: Slot [16] registered [ 0.260058] acpiphp: Slot [17] registered [ 0.262078] acpiphp: Slot [18] registered [ 0.263048] acpiphp: Slot [19] registered [ 0.264049] acpiphp: Slot [20] registered [ 0.264925] acpiphp: Slot [21] registered [ 0.266055] acpiphp: Slot [22] registered [ 0.267000] acpiphp: Slot [23] registered [ 0.267000] acpiphp: Slot [24] registered [ 0.269058] acpiphp: Slot [25] registered [ 0.269932] acpiphp: Slot [26] registered [ 0.270060] acpiphp: Slot [27] registered [ 0.270920] acpiphp: Slot [28] registered [ 0.272047] acpiphp: Slot [29] registered [ 0.272914] acpiphp: Slot [30] registered [ 0.274045] acpiphp: Slot [31] registered [ 0.275051] PCI host bridge to bus 0000:00 [ 0.277015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.278021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285011] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288147] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.289800] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294327] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.302010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.307798] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.311021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.313013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.316014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.318545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.321826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328447] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333008] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.343918] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.348007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.351955] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.355999] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.362010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.375015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.382337] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.389011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.395012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.410013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.419000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.425012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.430011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.447011] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.456564] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.462011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.469010] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.486015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.495241] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.498884] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.502873] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.516015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.521917] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.527017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.530719] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.541013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.551205] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.552262] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.554200] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.555220] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.557190] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.563033] iommu: Default domain type: Passthrough [ 0.564389] SCSI subsystem initialized [ 0.565062] ACPI: bus type USB registered [ 0.567085] usbcore: registered new interface driver usbfs [ 0.568042] usbcore: registered new interface driver hub [ 0.570057] usbcore: registered new device driver usb [ 0.572112] pps_core: LinuxPPS API ver. 1 registered [ 0.573005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.575021] PTP clock support registered [ 0.576111] EDAC MC: Ver: 3.0.0 [ 0.577354] PCI: Using ACPI for IRQ routing [ 0.579916] NetLabel: Initializing [ 0.581013] NetLabel: domain hash size = 128 [ 0.583009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.585068] NetLabel: unlabeled traffic allowed by default [ 0.587113] vgaarb: loaded [ 0.589123] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.590008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.593271] clocksource: Switched to clocksource kvm-clock [ 0.676368] VFS: Disk quotas dquot_6.6.0 [ 0.677355] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.678842] *** VALIDATE ramfs *** [ 0.679470] *** VALIDATE hugetlbfs *** [ 0.680412] pnp: PnP ACPI init [ 0.682369] pnp: PnP ACPI: found 6 devices [ 0.694945] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.696872] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.698127] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.699380] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.700795] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.702284] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.703963] NET: Registered protocol family 2 [ 0.705555] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.708607] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.710973] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.714816] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.716810] TCP: Hash tables configured (established 65536 bind 65536) [ 0.718649] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.720470] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.722150] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.723878] NET: Registered protocol family 1 [ 0.725427] RPC: Registered named UNIX socket transport module. [ 0.726661] RPC: Registered udp transport module. [ 0.727629] RPC: Registered tcp transport module. [ 0.728617] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.730031] NET: Registered protocol family 44 [ 0.731038] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.732312] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.733812] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.735494] PCI: CLS 0 bytes, default 64 [ 0.736514] Unpacking initramfs... [ 2.110917] debug: unmapping init [mem 0xffff9402bcc54000-0xffff9402bffbffff] [ 2.113414] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.114626] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.116276] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.569789] Initialise system trusted keyrings [ 2.571068] Key type blacklist registered [ 2.572391] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.581464] zbud: loaded [ 2.584837] *** VALIDATE nfs *** [ 2.586235] *** VALIDATE nfs4 *** [ 2.588045] pstore: using deflate compression [ 2.590917] Platform Keyring initialized [ 2.662692] NET: Registered protocol family 38 [ 2.664077] Key type asymmetric registered [ 2.665125] Asymmetric key parser 'x509' registered [ 2.666990] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.669146] io scheduler mq-deadline registered [ 2.670931] io scheduler kyber registered [ 2.672673] io scheduler bfq registered [ 2.674405] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.677351] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.680180] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.682995] ACPI: Power Button [PWRF] [ 2.688203] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.695816] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.710589] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.718151] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.729464] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.757937] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.790492] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.795351] Non-volatile memory driver v1.3 [ 2.796470] Linux agpgart interface v0.103 [ 2.821961] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.824900] vda: detected capacity change from 0 to 74690560 [ 2.840912] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.844011] vdb: detected capacity change from 0 to 1073741824 [ 2.860247] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.863306] vdc: detected capacity change from 0 to 2621440000 [ 2.878655] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.882043] vdd: detected capacity change from 0 to 2621440000 [ 2.899359] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.901365] vde: detected capacity change from 0 to 4294967296 [ 2.913445] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.916323] vdf: detected capacity change from 0 to 4294967296 [ 2.922181] libphy: Fixed MDIO Bus: probed [ 2.926147] usbcore: registered new interface driver usbserial_generic [ 2.928571] usbserial: USB Serial support registered for generic [ 2.931383] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.935099] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.936430] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.938332] mousedev: PS/2 mouse device common for all mice [ 2.941376] rtc_cmos 00:05: RTC can wake from S4 [ 2.941497] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.943872] rtc_cmos 00:05: registered as rtc0 [ 2.948551] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.952025] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.953561] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.955373] intel_pstate: CPU model not supported [ 2.960928] hid: raw HID events driver (C) Jiri Kosina [ 2.963389] usbcore: registered new interface driver usbhid [ 2.964664] usbhid: USB HID core driver [ 2.965634] drop_monitor: Initializing network drop monitor service [ 2.968452] Initializing XFRM netlink socket [ 2.970524] NET: Registered protocol family 10 [ 2.973543] Segment Routing with IPv6 [ 2.975418] NET: Registered protocol family 17 [ 2.977722] mpls_gso: MPLS GSO support [ 2.983846] RAS: Correctable Errors collector initialized. [ 2.986116] AVX version of gcm_enc/dec engaged. [ 2.987868] AES CTR mode by8 optimization enabled [ 3.043565] sched_clock: Marking stable (3043539861, 0)->(3813888496, -770348635) [ 3.046892] registered taskstats version 1 [ 3.048536] Loading compiled-in X.509 certificates [ 3.049784] zswap: loaded using pool lzo/zbud [ 3.069371] Key type big_key registered [ 3.080550] Key type encrypted registered [ 3.082291] ima: No TPM chip found, activating TPM-bypass! [ 3.086654] ima: Allocated hash algorithm: sha1 [ 3.088228] ima: No architecture policies found [ 3.089836] evm: Initialising EVM extended attributes: [ 3.091840] evm: security.selinux [ 3.093115] evm: security.ima [ 3.094144] evm: security.capability [ 3.095338] evm: HMAC attrs: 0x1 [ 3.097602] rtc_cmos 00:05: setting system clock to 2026-07-25 02:41:52 UTC (1784947312) [ 3.104136] debug: unmapping init [mem 0xffffffffa8203000-0xffffffffa83fffff] [ 3.106870] debug: unmapping init [mem 0xffffffffa6f82000-0xffffffffa7258fff] [ 3.117081] Write protecting the kernel read-only data: 28672k [ 3.120257] debug: unmapping init [mem 0xffffffffa5603000-0xffffffffa57fffff] [ 3.122909] debug: unmapping init [mem 0xffffffffa5f14000-0xffffffffa5ffffff] [ 3.154683] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.162810] systemd[1]: Detected virtualization kvm. [ 3.164609] systemd[1]: Detected architecture x86-64. [ 3.166369] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.193109] systemd[1]: No hostname configured. [ 3.194652] systemd[1]: Set hostname to . [ 3.196085] random: systemd: uninitialized urandom read (16 bytes read) [ 3.197949] systemd[1]: Initializing machine ID from random generator. [ 3.257375] random: ln: uninitialized urandom read (6 bytes read) [ 3.320875] random: systemd: uninitialized urandom read (16 bytes read) [ 3.323527] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.326708] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.332204] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.838587] device-mapper: uevent: version 1.0.3 [ 3.840876] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. [ 4.033592] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.516106] virtio_net virtio0 ens2: renamed from eth0 [ 4.519924] scsi host0: ata_piix [ 4.532912] scsi host1: ata_piix [ 4.604650] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.606877] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.168563] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.601706] random: crng init done [ 9.602505] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.747832] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.614564] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.766330] SELinux: Disabled at runtime. [ 10.816940] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.822930] systemd[1]: Detected virtualization kvm. [ 10.824386] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.137699] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.140678] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.144868] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.147976] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.150976] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.157725] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.160173] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [[ 11.213031] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 11.496076] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 11.776846] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.803638] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.878295] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.908933] EDAC sbridge: Ver: 1.1.2 [ 12.869429] Key type dns_resolver registered [ 13.114051] NFS: Registering the id_resolver key type [ 13.115218] Key type id_resolver registered [ 13.116017] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg424-server login: [ 32.930355] libcfs: loading out-of-tree module taints kernel. [ 32.950649] Key type ._llcrypt registered [ 32.952301] Key type .llcrypt registered [ 32.989262] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 39.617162] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 40.104229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 40.110743] alg: No test for adler32 (adler32-zlib) [ 41.102803] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 41.393320] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 43.008080] Key type lgssc registered [ 43.574848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 49.450198] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 62.152446] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 66.213402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 66.219921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 67.296937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 67.308141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 67.348348] Lustre: lustre-MDT0000: new disk, initializing [ 67.369959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 67.377359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 68.543558] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 73.551246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 73.591292] Lustre: 6469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 73.602249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 73.605140] Lustre: Skipped 1 previous similar message [ 73.647881] Lustre: lustre-MDT0001: new disk, initializing [ 73.672610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 73.681759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 73.685596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 75.065318] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 77.288835] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 80.241400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 80.326719] Lustre: lustre-OST0000: new disk, initializing [ 80.328191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 80.329853] Lustre: 8399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 80.344343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 81.133995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 81.136318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 81.160231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 82.183166] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 87.346604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 87.390794] Lustre: lustre-OST0001: new disk, initializing [ 87.392082] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 87.394307] Lustre: 9477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 87.411405] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 89.157820] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 89.582604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 89.585317] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 89.607646] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 94.689962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 97.204749] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 99.338523] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing check_logdir /tmp/testlogs/ [ 100.720123] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing yml_node [ 102.144757] Lustre: DEBUG MARKER: Client: 2.17.54.203 [ 102.924101] Lustre: DEBUG MARKER: MDS: 2.17.54.203 [ 103.755500] Lustre: DEBUG MARKER: OSS: 2.17.54.203 [ 104.236096] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 24 22:43:33 EDT 2026 [ 109.737099] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 110.282181] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 124.896788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 124.899740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 124.903811] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 125.408698] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 125.409329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 125.410910] Lustre: Skipped 1 previous similar message [ 125.414813] Lustre: Skipped 1 previous similar message [ 128.154252] Lustre: server umount lustre-MDT0000 complete [ 129.252724] LustreError: 6463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947438 with bad export cookie 3269318921390048697 [ 129.253790] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 129.256020] LustreError: 6463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 130.528897] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 130.533466] Lustre: Skipped 1 previous similar message [ 130.535459] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 130.538273] Lustre: Skipped 3 previous similar messages [ 134.560700] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 134.562944] Lustre: Skipped 1 previous similar message [ 135.391349] Lustre: server umount lustre-MDT0001 complete [ 142.717810] Lustre: server umount lustre-OST0000 complete [ 158.176084] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 158.223886] Lustre: server umount lustre-OST0001 complete [ 160.163614] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 163.146458] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 174.306606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.396562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.409367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.443126] Lustre: lustre-MDT0000: new disk, initializing [ 174.462678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.469493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.724268] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 179.693354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.734404] Lustre: 15015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.847703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.164292] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 184.791656] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 186.554483] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 188.726298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.840616] Lustre: lustre-OST0000: new disk, initializing [ 188.842401] Lustre: Skipped 1 previous similar message [ 188.844478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 188.847835] Lustre: Skipped 2 previous similar messages [ 188.849972] Lustre: 16982:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 188.879582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.882699] Lustre: Skipped 1 previous similar message [ 190.013179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 190.015853] Lustre: Skipped 1 previous similar message [ 190.017966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 190.061043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.700248] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 194.277267] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 195.663426] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 195.735325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 197.160510] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 197.238288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 200.166639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 200.167174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 200.168364] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 200.179182] Lustre: Skipped 2 previous similar messages [ 203.988563] Lustre: server umount lustre-OST0000 complete [ 205.793153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 205.796746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 210.401740] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 210.404860] Lustre: Skipped 3 previous similar messages [ 211.486063] Lustre: server umount lustre-MDT0000 complete [ 212.736427] LustreError: 15009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947522 with bad export cookie 3269318921390050160 [ 212.740240] LustreError: 15009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 212.740575] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 212.852814] Lustre: server umount lustre-MDT0001 complete [ 214.714781] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:45:23 (1784947523) [ 220.788976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.936637] LustreError: 19850:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 220.944255] LustreError: 19850:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 220.958962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.249038] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 225.034087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.477089] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 228.812968] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 230.629143] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 231.330391] Lustre: Failing over lustre-MDT0000 [ 231.426080] Lustre: server umount lustre-MDT0000 complete [ 235.425900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 235.426631] LustreError: 19845:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 235.429538] Lustre: Skipped 2 previous similar messages [ 240.609317] LustreError: 19875:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 240.613735] LustreError: 19875:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 244.290563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.328696] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 245.637442] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 249.250802] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 249.827389] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 249.830024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.204.124@tcp (at 192.168.204.124@tcp) [ 249.836055] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 251.259048] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 251.261109] Lustre: Skipped 1 previous similar message [ 257.435661] Lustre: server umount lustre-MDT0000 complete [ 258.661192] LustreError: 19833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947568 with bad export cookie 3269318921390051574 [ 258.663897] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 258.666012] LustreError: 19833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 258.777361] Lustre: server umount lustre-MDT0001 complete [ 261.545049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 261.692129] LustreError: 23183:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.715970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 261.718625] Lustre: Skipped 2 previous similar messages [ 263.018248] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 265.791062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 267.200963] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 269.549704] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 271.385610] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 272.086656] Lustre: Failing over lustre-MDT0000 [ 272.092270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 272.095807] Lustre: Skipped 3 previous similar messages [ 272.177898] Lustre: server umount lustre-MDT0000 complete [ 276.193693] LustreError: 23924:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.200565] LustreError: 23924:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 284.988333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 285.032568] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.422695] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 290.097824] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 290.274400] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 290.277563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.204.124@tcp (at 192.168.204.124@tcp) [ 290.282009] Lustre: Skipped 1 previous similar message [ 290.287106] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 295.393588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 295.398342] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 295.401196] Lustre: Skipped 4 previous similar messages [ 298.203187] Lustre: server umount lustre-MDT0000 complete [ 299.428308] LustreError: 23163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947608 with bad export cookie 3269318921390052512 [ 299.431465] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.432959] LustreError: 23163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 299.554572] Lustre: server umount lustre-MDT0001 complete [ 302.466184] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:46:51 (1784947611) [ 307.476881] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 310.410886] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 322.627779] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 326.446797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 326.539169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 326.550174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 326.589821] Lustre: lustre-MDT0000: new disk, initializing [ 326.614801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.617414] Lustre: Skipped 2 previous similar messages [ 326.623427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 328.015235] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 332.215618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.251683] Lustre: 29541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 332.339831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 333.679463] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 337.382650] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 339.183180] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 341.494874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 341.662899] Lustre: 31515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 343.174714] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 343.199300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 343.207603] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 343.634468] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 347.131240] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 348.948937] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 353.767492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 353.768415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 353.772251] Lustre: Skipped 3 previous similar messages [ 353.775391] LustreError: Skipped 1 previous similar message [ 358.444668] Lustre: server umount lustre-OST0000 complete [ 362.977207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 362.981048] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 362.983287] Lustre: Skipped 4 previous similar messages [ 365.919725] Lustre: server umount lustre-MDT0000 complete [ 367.095683] LustreError: 29578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947676 with bad export cookie 3269318921390052918 [ 367.099624] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.100422] LustreError: 29578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 367.286867] Lustre: server umount lustre-MDT0001 complete [ 370.679843] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 371.654332] Key type lgssc unregistered [ 371.785360] LNet: 33464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.788201] LNetError: 33464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 371.797306] LNet: Removed LNI 192.168.204.124@tcp [ 372.142122] Key type .llcrypt unregistered [ 372.143309] Key type ._llcrypt unregistered [ 380.864561] Key type ._llcrypt registered [ 380.865550] Key type .llcrypt registered [ 380.900223] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 381.204201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 381.241568] alg: No test for adler32 (adler32-zlib) [ 382.100199] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 382.191670] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 383.776127] Key type lgssc registered [ 384.112455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 386.550986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 386.558563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.696802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.010763] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 391.709136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.817806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 393.142652] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 395.520555] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 397.219425] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 399.270032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 399.394802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.303624] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 402.473739] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 403.653120] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 418.335441] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:48:47 (1784947727) [ 419.005243] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 419.602528] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 421.793347] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:48:50 (1784947730) [ 424.537066] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:48:53 (1784947733) [ 424.847754] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 427.486065] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 427.488711] Lustre: Skipped 2 previous similar messages [ 427.992347] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:48:56 (1784947736) [ 434.588785] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 434.590957] Lustre: Skipped 10 previous similar messages [ 435.413861] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:49:04 (1784947744) [ 436.320192] Lustre: *** cfs_fail_loc=131b, val=2*** [ 437.437197] Lustre: *** cfs_fail_loc=131b, val=2*** [ 439.996261] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:49:08 (1784947748) [ 443.936563] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 443.939742] Lustre: Skipped 7 previous similar messages [ 444.490978] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:49:13 (1784947753) [ 448.488535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 448.488883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 448.490115] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 448.491809] LustreError: Skipped 1 previous similar message [ 448.497351] Lustre: Skipped 1 previous similar message [ 453.603440] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 453.606700] Lustre: Skipped 2 previous similar messages [ 454.412913] Lustre: server umount lustre-OST0000 complete [ 458.725869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 458.726040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 458.726983] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 458.738975] Lustre: Skipped 1 previous similar message [ 461.845710] Lustre: server umount lustre-MDT0000 complete [ 462.954199] LustreError: 35587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947772 with bad export cookie 12818447634249390528 [ 462.956785] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.958251] LustreError: 35587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 463.074430] Lustre: server umount lustre-MDT0001 complete [ 466.417275] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 467.392931] Key type lgssc unregistered [ 467.525376] LNet: 42701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.528637] LNetError: 42701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.537302] LNet: Removed LNI 192.168.204.124@tcp [ 467.830203] Key type .llcrypt unregistered [ 467.831282] Key type ._llcrypt unregistered [ 473.676667] Key type ._llcrypt registered [ 473.678101] Key type .llcrypt registered [ 473.709960] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 479.635277] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 479.999560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 480.021347] alg: No test for adler32 (adler32-zlib) [ 480.884509] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 480.968527] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 482.552187] Key type lgssc registered [ 482.884988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 493.325328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 493.332049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.427288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 494.438261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 494.472084] Lustre: lustre-MDT0000: new disk, initializing [ 494.492957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.498558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 495.777820] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 499.788608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.829128] Lustre: 45826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 499.855456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 499.857821] Lustre: Skipped 1 previous similar message [ 499.900666] Lustre: lustre-MDT0001: new disk, initializing [ 499.934034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 499.946813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 499.950435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 501.221345] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 504.758293] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 506.516791] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 508.788912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 508.913234] Lustre: lustre-OST0000: new disk, initializing [ 508.915271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 508.917979] Lustre: 47792:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 508.950048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.934295] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 513.544055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 513.547881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 513.585446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 514.678388] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.003577] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 516.069323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.447861] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 517.522519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 518.627071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 518.628222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 518.629624] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 518.631395] LustreError: Skipped 1 previous similar message [ 518.636872] Lustre: Skipped 1 previous similar message [ 520.205614] Lustre: server umount lustre-OST0000 complete [ 523.746292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 523.746951] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 523.752121] Lustre: Skipped 1 previous similar message [ 523.754268] Lustre: Skipped 1 previous similar message [ 527.572107] Lustre: server umount lustre-MDT0000 complete [ 528.712062] LustreError: 45816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947838 with bad export cookie 11537603775022687543 [ 528.714613] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.716816] LustreError: 45816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 528.827418] Lustre: server umount lustre-MDT0001 complete [ 531.799590] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:50:40 (1784947840) [ 533.993938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.163098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.472276] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 538.258835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.387391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 539.726636] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 542.062473] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 543.832609] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 546.029058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 548.127948] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 550.486770] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 562.705244] LustreError: 52797:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 563.005466] Lustre: Modifying parameter general.timeout=1 in log params [ 564.170054] Lustre: Modifying parameter general.timeout=5 in log params [ 564.171823] Lustre: Skipped 3 previous similar messages [ 566.384182] Lustre: Modifying parameter general.timeout=13 in log params [ 566.386616] Lustre: Skipped 7 previous similar messages [ 570.625832] Lustre: Modifying parameter general.timeout=28 in log params [ 570.627415] Lustre: Skipped 14 previous similar messages [ 578.915501] Lustre: Modifying parameter general.timeout=57 in log params [ 578.918187] Lustre: Skipped 28 previous similar messages [ 594.930161] Lustre: Modifying parameter general.timeout=113 in log params [ 594.932942] Lustre: Skipped 55 previous similar messages [ 602.864322] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:51:51 (1784947911) [ 603.793718] Lustre: 59688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 604.983034] Lustre: 59884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 604.988832] Lustre: 59884:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 614.813248] Lustre: 61514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 614.817698] Lustre: 61514:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 3 previous similar messages [ 614.821135] LustreError: 61514:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 616.074735] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:52:05 (1784947925) [ 620.513256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 620.516979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 620.523264] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 620.525480] Lustre: Skipped 1 previous similar message [ 621.522170] Lustre: server umount lustre-MDT0000 complete [ 622.659697] LustreError: 52089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947932 with bad export cookie 11537603775022688425 [ 622.664555] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.668495] LustreError: 52089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 623.073629] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 623.074277] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 623.076598] Lustre: Skipped 1 previous similar message [ 623.080266] LustreError: 50246:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.084468] LustreError: 50246:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 627.105214] LustreError: 58431:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.105367] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 628.840479] Lustre: server umount lustre-MDT0001 complete [ 630.115437] Lustre: server umount lustre-OST0000 complete [ 638.495587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.604400] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 638.614126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 638.617235] Lustre: Skipped 39 previous similar messages [ 638.700904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.703088] Lustre: Skipped 1 previous similar message [ 640.059410] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 642.838931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.873705] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 642.885407] Lustre: 64039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 644.306804] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 646.608089] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 648.403213] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 650.598832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 650.675186] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 652.696949] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 655.135553] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 655.667037] Lustre: 66603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 658.917118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 658.918800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 658.919645] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 658.922398] Lustre: Skipped 1 previous similar message [ 659.538945] Lustre: server umount lustre-OST0000 complete [ 663.521073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 663.523489] LustreError: Skipped 1 previous similar message [ 666.836972] Lustre: server umount lustre-MDT0000 complete [ 668.035268] LustreError: 65897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947977 with bad export cookie 11537603775022690651 [ 668.038866] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.039642] LustreError: 65897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 668.159444] Lustre: server umount lustre-MDT0001 complete [ 671.673921] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 672.587153] Key type lgssc unregistered [ 672.712504] LNet: 67804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.715697] LNetError: 67804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.727385] LNet: Removed LNI 192.168.204.124@tcp [ 673.009700] Key type .llcrypt unregistered [ 673.010660] Key type ._llcrypt unregistered [ 681.602786] Key type ._llcrypt registered [ 681.604013] Key type .llcrypt registered [ 681.637867] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 682.002324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.064740] alg: No test for adler32 (adler32-zlib) [ 682.917630] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 683.011547] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 684.592198] Key type lgssc registered [ 684.902050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 687.297297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 687.302917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.451281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.687414] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 692.250637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.421613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 693.644759] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 695.807851] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 697.464586] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 699.498298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 699.683104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.539508] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 703.812428] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 717.071577] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 719.339718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 719.344813] Lustre: Skipped 2 previous similar messages [ 719.344906] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 719.349258] Lustre: Skipped 2 previous similar messages [ 724.451652] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 724.454246] Lustre: Skipped 2 previous similar messages [ 725.468814] Lustre: server umount lustre-MDT0000 complete [ 726.633310] LustreError: 71900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784948036 with bad export cookie 11935183876745713334 [ 726.636846] LustreError: 71900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 726.637370] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.641416] LustreError: 69187:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 726.745214] Lustre: server umount lustre-MDT0001 complete [ 738.137852] Lustre: server umount lustre-OST0000 complete [ 746.142749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.228522] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 746.237963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 746.324920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 747.492214] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 750.095890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.133165] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 750.145313] Lustre: 74080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 751.680643] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 754.111431] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 755.940753] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 758.162990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 758.235387] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 758.371084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.373299] Lustre: Skipped 1 previous similar message [ 760.305433] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 762.682141] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 768.619073] Lustre: 76655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 769.271097] Lustre: Modifying parameter general.jobid_var=disable in log params [ 772.068983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 772.069688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 772.072431] LustreError: Skipped 1 previous similar message [ 772.077136] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 772.080895] Lustre: Skipped 1 previous similar message [ 776.661141] Lustre: server umount lustre-OST0000 complete [ 781.281319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 781.285261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 781.289443] Lustre: Skipped 1 previous similar message [ 781.291562] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 784.086098] Lustre: server umount lustre-MDT0000 complete [ 785.340542] LustreError: 74119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784948094 with bad export cookie 11935183876745714125 [ 785.343839] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.346720] LustreError: 74119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 785.473272] Lustre: server umount lustre-MDT0001 complete [ 789.337155] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 790.331327] Key type lgssc unregistered [ 790.455366] LNet: 77862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.458594] LNetError: 77862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.466251] LNet: Removed LNI 192.168.204.124@tcp [ 790.741143] Key type .llcrypt unregistered [ 790.742164] Key type ._llcrypt unregistered [ 798.931917] Key type ._llcrypt registered [ 798.932877] Key type .llcrypt registered [ 798.965278] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 799.250661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.293632] alg: No test for adler32 (adler32-zlib) [ 800.136767] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 800.219417] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 801.800180] Key type lgssc registered [ 802.091484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 804.442438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 804.448051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.580735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.812176] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 809.340630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.511580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 810.759413] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 812.935401] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 814.569982] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 816.587105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 816.795099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 818.476521] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 820.646739] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 833.274362] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 833.810406] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 833.812105] Lustre: Skipped 1 previous similar message [ 834.863065] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 834.865091] Lustre: Skipped 3 previous similar messages [ 837.069710] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 837.072412] Lustre: Skipped 7 previous similar messages [ 841.173902] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 841.176082] Lustre: Skipped 14 previous similar messages [ 849.439994] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 849.442169] Lustre: Skipped 29 previous similar messages [ 862.908318] Lustre: 87333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 863.662706] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:56:12 (1784948172) [ 864.114733] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 864.606818] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:56:13 (1784948173) [ 866.276104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 866.276545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 866.277215] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 866.279440] Lustre: Skipped 1 previous similar message [ 871.394021] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 871.396772] Lustre: Skipped 2 previous similar messages [ 871.567624] Lustre: server umount lustre-OST0000 complete [ 876.004977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 876.007461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 876.012433] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 881.634416] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 881.636769] Lustre: Skipped 2 previous similar messages [ 886.754029] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 886.756133] Lustre: Skipped 1 previous similar message [ 887.264124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 887.324164] Lustre: server umount lustre-MDT0000 complete [ 888.479063] LustreError: 81859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784948197 with bad export cookie 9796813325898983821 [ 888.481527] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 888.482715] LustreError: 81859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 888.600799] Lustre: server umount lustre-MDT0001 complete [ 892.101983] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 893.044184] Key type lgssc unregistered [ 893.160340] LNet: 88580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.162412] LNetError: 88580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.171276] LNet: Removed LNI 192.168.204.124@tcp [ 893.452667] Key type .llcrypt unregistered [ 893.453797] Key type ._llcrypt unregistered [ 894.762022] Key type ._llcrypt registered [ 894.762960] Key type .llcrypt registered [ 894.794308] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 896.083799] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules [ 896.404482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 896.480226] alg: No test for adler32 (adler32-zlib) [ 897.321271] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 897.395935] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 898.968178] Key type lgssc registered [ 899.356477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 899.687798] LustreError: 89733:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 906.953388] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 909.691055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 920.928295] LustreError: 89733:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 920.936676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.120248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.521166] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 926.393980] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:57:15 (1784948235) [ 927.934638] LustreError: 90795:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 927.938184] LustreError: 90795:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 927.940362] LustreError: 90795:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 928.044705] Lustre: server umount lustre-MDT0000 complete [ 932.258885] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 933.225963] Key type lgssc unregistered [ 933.337367] LNet: 92289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.339787] LNetError: 92289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.350262] LNet: Removed LNI 192.168.204.124@tcp [ 933.625896] Key type .llcrypt unregistered [ 933.627057] Key type ._llcrypt unregistered [ 940.215138] Key type ._llcrypt registered [ 940.216611] Key type .llcrypt registered [ 940.250259] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 940.524827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.564459] alg: No test for adler32 (adler32-zlib) [ 941.404144] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 941.483133] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 943.056107] Key type lgssc registered [ 943.362308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 945.812306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 945.819692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.007635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.441508] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 951.425289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.681443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 953.164769] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 955.737126] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 957.660395] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 959.868853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.073744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.021685] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 964.381300] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 995.701257] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:58:24 (1784948304) [ 996.269899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 996.845108] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:58:25 (1784948305) [ 997.856952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 997.859366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 997.863515] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 998.369757] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 998.371314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1003.490112] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 1003.492950] Lustre: Skipped 4 previous similar messages [ 1003.983143] Lustre: server umount lustre-MDT0000 complete [ 1005.130991] LustreError: 95330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784948314 with bad export cookie 10142847775254614478 [ 1005.133754] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1005.134097] LustreError: 95330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1005.230593] Lustre: server umount lustre-MDT0001 complete [ 1016.641118] Lustre: server umount lustre-OST0000 complete [ 1020.358540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.519396] LustreError: 97774:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.527429] LustreError: 97774:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1020.573496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.918924] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1024.726513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.959511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.248779] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1028.557186] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1030.337485] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1034.520688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1034.592954] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1034.596793] LustreError: 97762:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1034.607436] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1034.611984] LustreError: 99855:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1034.615855] LustreError: 99855:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1034.619113] LustreError: 99855:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1034.653277] Lustre: server umount lustre-OST0000 complete [ 1034.655187] LustreError: 99855:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1037.322849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.386556] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1037.389891] LustreError: 97762:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1037.398911] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1037.402795] LustreError: 100316:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1037.405716] LustreError: 100316:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1037.408324] LustreError: 100316:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1037.431945] Lustre: server umount lustre-OST0000 complete [ 1037.433794] LustreError: 100316:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1040.331600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1040.396573] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1040.398530] Lustre: Found index 0 for lustre-OST0000, updating log [ 1040.400815] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1040.423542] Lustre: lustre-OST0000: new disk, initializing [ 1040.425790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1040.429121] Lustre: 100849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1040.524286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.276591] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1043.372407] Lustre: Failing over lustre-OST0000 [ 1043.411643] Lustre: server umount lustre-OST0000 complete [ 1045.473084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 1045.476064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1045.480129] Lustre: Skipped 1 previous similar message [ 1045.481808] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 1050.081509] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 1050.084092] Lustre: Skipped 2 previous similar messages [ 1050.645240] Lustre: server umount lustre-MDT0000 complete [ 1051.805258] LustreError: 97756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784948361 with bad export cookie 10142847775254617208 [ 1051.808294] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1051.809341] LustreError: 97756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1051.931789] Lustre: server umount lustre-MDT0001 complete [ 1054.570701] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:59:23 (1784948363) [ 1059.790459] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1063.338138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.484294] LustreError: 103186:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.507909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.855427] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1067.678734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.277457] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1070.217350] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1072.379621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1074.635433] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1077.542405] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.573854] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1077.612330] Lustre: lustre-OST0001: new disk, initializing [ 1077.614513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1077.618866] Lustre: 105505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1079.772496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1079.776679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1079.781102] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1079.816596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1082.955252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1089.632425] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1093.089198] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 1093.092026] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1093.096323] Lustre: Skipped 1 previous similar message [ 1093.098333] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 1105.376154] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1105.447223] Lustre: server umount lustre-MDT0001 complete [ 1106.173327] LustreError: 103923:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.751353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.787913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1108.789706] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1108.806061] Lustre: 103176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1109.107572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1109.111243] Lustre: Skipped 3 previous similar messages [ 1110.569193] LustreError: 107220:0:(obd_mount.c:230:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1110.572906] LustreError: 107220:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1110.577515] LustreError: 103179:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1110.615913] LustreError: 107234:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.124@tcp: cfg command failed: rc = -17 [ 1110.619366] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1110.619366] [ 1110.715287] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1114.090781] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:00:22 (1784948422) [ 1114.602511] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.204.124@tcp (at 192.168.204.124@tcp) [ 1119.868017] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1125.996591] Lustre: 109109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1133.877338] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1135.019071] Lustre: 110240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1180.640933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 1180.644155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1180.648324] Lustre: Skipped 2 previous similar messages [ 1180.650147] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 1180.652996] Lustre: Skipped 11 previous similar messages [ 1186.202887] Lustre: server umount lustre-MDT0000 complete [ 1186.273439] LustreError: 110684:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1186.281918] LustreError: 110684:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 1187.525211] LustreError: 103168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784948496 with bad export cookie 10142847775254618076 [ 1187.527055] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.531156] LustreError: 103168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1187.652143] Lustre: server umount lustre-MDT0001 complete [ 1199.629941] Lustre: server umount lustre-OST0000 complete [ 1211.226914] Lustre: server umount lustre-OST0001 complete [ 1213.087558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1216.214757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1219.015316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1222.186804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1228.612147] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1232.226978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.233894] Lustre: lustre-MDT0000: reset Object Index mappings [ 1232.387418] LustreError: 114170:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.411609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.700052] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1236.405758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.413524] Lustre: lustre-MDT0001: reset Object Index mappings [ 1238.033637] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1238.958393] Lustre: 115318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1241.067905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1242.340583] LustreError: 115672:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.345862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1243.138854] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1245.728399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.956229] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1246.957276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1246.990292] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1247.741431] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1250.748175] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1251.854442] Lustre: 117193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1266.466650] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:02:55 (1784948575) [ 1297.889452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 1297.892175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1297.895936] Lustre: Skipped 3 previous similar messages [ 1297.897818] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 1297.900279] Lustre: Skipped 4 previous similar messages [ 1301.621521] Lustre: server umount lustre-MDT0000 complete [ 1302.784197] LustreError: 117195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784948612 with bad export cookie 10142847775254664843 [ 1302.786572] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.789158] LustreError: 117195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1303.521609] LustreError: 114167:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.526331] LustreError: 114167:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1317.344148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1317.444712] Lustre: server umount lustre-MDT0001 complete [ 1324.804614] Lustre: server umount lustre-OST0000 complete [ 1340.384183] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1340.459707] Lustre: server umount lustre-OST0001 complete [ 1342.591479] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 1345.808174] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1357.830042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.925923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1357.936158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1357.974454] Lustre: lustre-MDT0000: new disk, initializing [ 1357.999726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1359.517909] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1362.514427] Lustre: server umount lustre-MDT0000 complete [ 1365.641835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.764984] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1365.769720] Lustre: 121454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1365.774147] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1365.859202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.862407] Lustre: Skipped 4 previous similar messages [ 1367.205163] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1369.040322] Lustre: server umount lustre-MDT0000 complete [ 1371.842382] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:04:40 (1784948680) [ 1372.406135] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1373.012709] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:04:41 (1784948681) [ 1378.706765] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1382.100500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.641309] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1386.499967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.542463] Lustre: 123505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1386.564754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1386.567153] Lustre: Skipped 1 previous similar message [ 1386.603167] Lustre: lustre-MDT0001: new disk, initializing [ 1386.647782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1386.651548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1388.022619] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1390.393147] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1392.627922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.767377] Lustre: lustre-OST0000: new disk, initializing [ 1392.769461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1392.772172] Lustre: 125080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1394.052225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1394.056399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1394.103479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1394.854675] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1399.025083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.092437] Lustre: 125951:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1400.601296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1400.605063] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1400.649175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1401.056145] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1405.436435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1406.563707] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1556.870774] Lustre: 132321:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1556.874861] Lustre: 132321:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1556.878524] Lustre: 132321:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1556.881119] Lustre: 132321:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1556.884597] Lustre: 132321:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1556.887894] Lustre: 132321:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1639.025158] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:09:07 (1784948947) [ 1653.217233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 1653.219720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1653.222979] Lustre: Skipped 5 previous similar messages [ 1653.224605] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 1653.228420] Lustre: Skipped 10 previous similar messages [ 1656.695229] Lustre: server umount lustre-MDT0000 complete [ 1657.796308] LustreError: 123497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784948967 with bad export cookie 10142847775254717875 [ 1657.798367] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.799330] LustreError: 123497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1659.874012] LustreError: 126476:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.883894] LustreError: 126476:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 1672.160086] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1672.265701] Lustre: server umount lustre-MDT0001 complete [ 1679.696414] Lustre: server umount lustre-OST0000 complete [ 1687.385783] Lustre: server umount lustre-OST0001 complete [ 1689.464903] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 1692.808370] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1704.991274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.080797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1705.089947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1705.091624] Lustre: Skipped 1 previous similar message [ 1705.126676] Lustre: lustre-MDT0000: new disk, initializing [ 1705.127973] Lustre: Skipped 1 previous similar message [ 1705.148055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.149934] Lustre: Skipped 4 previous similar messages [ 1705.154382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1706.457967] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1710.461314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.501603] Lustre: 135339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1710.610678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1712.030073] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1715.664379] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1717.462967] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1719.684559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.789749] Lustre: 137302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1721.531056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1721.534036] Lustre: Skipped 1 previous similar message [ 1721.537082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1721.569973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1721.705041] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1725.570721] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1728.650390] Lustre: lustre-MDD0000: changelog on [ 1729.560528] Lustre: lustre-MDD0001: changelog on [ 2093.134242] Lustre: 135343:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2122.990715] Lustre: lustre-MDD0001: changelog off [ 2124.127701] Lustre: lustre-MDD0000: changelog off [ 2126.309810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2126.309855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2126.314710] LustreError: Skipped 1 previous similar message [ 2126.321651] Lustre: Skipped 5 previous similar messages [ 2126.324183] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2126.327931] Lustre: Skipped 15 previous similar messages [ 2131.805205] Lustre: server umount lustre-OST0000 complete [ 2147.296083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2147.396133] Lustre: server umount lustre-MDT0000 complete [ 2148.592344] LustreError: 136131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784949457 with bad export cookie 10142847775255482842 [ 2148.595697] LustreError: 136131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2148.596046] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2148.728980] Lustre: server umount lustre-MDT0001 complete [ 2152.360065] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2153.355646] Key type lgssc unregistered [ 2153.479371] LNet: 146092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.481854] LNetError: 146092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.488397] LNet: Removed LNI 192.168.204.124@tcp [ 2153.789125] Key type .llcrypt unregistered [ 2153.791039] Key type ._llcrypt unregistered [ 2159.623495] Key type ._llcrypt registered [ 2159.624527] Key type .llcrypt registered [ 2159.657200] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 2165.378625] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2165.675223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2165.724496] alg: No test for adler32 (adler32-zlib) [ 2166.575342] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2166.657622] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2168.232103] Key type lgssc registered [ 2168.553793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2179.862030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2179.868107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.951924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2180.964544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2181.002907] Lustre: lustre-MDT0000: new disk, initializing [ 2181.022703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.029546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2182.304934] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2186.949510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.024286] Lustre: 149265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2187.066778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2187.070368] Lustre: Skipped 1 previous similar message [ 2187.112492] Lustre: lustre-MDT0001: new disk, initializing [ 2187.157270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.176155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2187.182796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2188.820772] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2192.843927] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2195.118583] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2197.705086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2197.851553] Lustre: lustre-OST0000: new disk, initializing [ 2197.854402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2197.858894] Lustre: 151230:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2197.903254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.078931] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2203.787117] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2204.152741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2204.156042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2204.190971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2210.989749] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_module obdecho/obdecho [ 2211.538470] LustreError: 152236:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2211.541181] LustreError: 152236:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2212.080118] LustreError: 152332:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 2217.953293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2217.956311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2217.963796] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2219.490261] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2219.491553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2219.494562] Lustre: Skipped 1 previous similar message [ 2219.501237] Lustre: Skipped 1 previous similar message [ 2221.208837] Lustre: server umount lustre-MDT0000 complete [ 2222.295049] LustreError: 151229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784949531 with bad export cookie 6072905225282119366 [ 2222.297375] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.298209] LustreError: 151229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2222.399289] Lustre: server umount lustre-MDT0001 complete [ 2232.789262] Lustre: server umount lustre-OST0000 complete [ 2239.453962] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 2242.683728] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2254.937154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.014106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2255.023396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2255.057538] Lustre: lustre-MDT0000: new disk, initializing [ 2255.077772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.083158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2256.336692] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2260.232170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.268260] Lustre: 156041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2260.292941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2260.294546] Lustre: Skipped 1 previous similar message [ 2260.327764] Lustre: lustre-MDT0001: new disk, initializing [ 2260.357104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.370446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2260.373680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2261.587162] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2265.421002] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2267.593557] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2270.068587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2270.182699] Lustre: lustre-OST0000: new disk, initializing [ 2270.186256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2270.189718] Lustre: 158006:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2270.231555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.510308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2272.514572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2272.545632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2272.864414] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2277.188942] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2278.591220] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2278.694699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2280.382227] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2280.491401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2282.985415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2282.985895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2282.988182] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2282.988190] Lustre: Skipped 1 previous similar message [ 2282.994747] LustreError: Skipped 1 previous similar message [ 2283.003404] Lustre: Skipped 1 previous similar message [ 2287.501236] Lustre: server umount lustre-OST0000 complete [ 2291.169313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2291.174124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2291.181656] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2291.183670] Lustre: Skipped 1 previous similar message [ 2294.747370] Lustre: server umount lustre-MDT0000 complete [ 2296.023127] LustreError: 156033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784949605 with bad export cookie 6072905225282120311 [ 2296.029558] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2296.031225] LustreError: 156033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2296.150197] Lustre: server umount lustre-MDT0001 complete [ 2297.437922] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:20:06 (1784949606) [ 2298.073414] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2298.897867] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:20:07 (1784949607) [ 2301.672201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.843690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.122330] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2305.884254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.327434] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2310.197907] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2312.387116] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2314.742125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2317.128771] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2319.565315] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2332.644305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2332.644907] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2332.648548] Lustre: Skipped 2 previous similar messages [ 2332.652399] Lustre: Skipped 4 previous similar messages [ 2338.796131] Lustre: server umount lustre-MDT0000 complete [ 2340.094579] LustreError: 161144:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.104215] LustreError: 161144:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2340.215770] LustreError: 160400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784949649 with bad export cookie 6072905225282121179 [ 2340.223226] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2340.224266] LustreError: 160400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2342.881636] LustreError: 162905:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.882139] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2342.898046] Lustre: Skipped 1 previous similar message [ 2342.900951] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 2342.906499] Lustre: Skipped 4 previous similar messages [ 2345.213148] LustreError: 160413:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.438591] Lustre: server umount lustre-MDT0001 complete [ 2349.204432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2353.769058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.864713] LustreError: 164276:0:(llog.c:1655:llog_backup()) MGC192.168.204.124@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2353.867707] Lustre: 164276:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.124@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2364.733730] LustreError: 164281:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.744898] LustreError: 164281:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2370.529830] LustreError: 164281:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.540621] LustreError: 164281:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2379.746175] LustreError: 164282:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.757211] LustreError: 164282:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 6 previous similar messages [ 2396.413246] LustreError: 164283:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.417975] LustreError: 164283:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 14 previous similar messages [ 2415.072436] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2415.076893] Lustre: 164276:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2430.433842] LustreError: 164283:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.442881] LustreError: 164283:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 19 previous similar messages [ 2456.046748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2456.051716] Lustre: Skipped 2 previous similar messages [ 2457.671783] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2460.135683] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.204.124@tcp (at 192.168.204.124@tcp) [ 2461.154043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.705033] LustreError: 147492:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000028cc4453 x1871655181853696/t0(0) o250->MGC192.168.204.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2486.334645] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2489.383556] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2490.026261] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2490.352468] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.204.124@tcp (at 192.168.204.124@tcp) [ 2491.912585] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2492.426862] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2500.581921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2500.583100] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2500.590178] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2500.595555] Lustre: Skipped 1 previous similar message [ 2500.601947] Lustre: Skipped 4 previous similar messages [ 2512.352086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2512.409957] Lustre: server umount lustre-MDT0000 complete [ 2513.689131] LustreError: 160400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784949823 with bad export cookie 6072905225282127654 [ 2513.691919] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2513.695265] LustreError: 160400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2513.812991] Lustre: server umount lustre-MDT0001 complete [ 2525.245779] Lustre: server umount lustre-OST0000 complete [ 2533.195446] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 2537.284894] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2552.377411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.500876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2552.515654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2552.565399] Lustre: lustre-MDT0000: new disk, initializing [ 2552.590558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.593609] Lustre: Skipped 1 previous similar message [ 2552.600123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2554.350186] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2559.476796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.540137] Lustre: 169355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2559.657341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2561.167817] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2565.731540] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2567.733736] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2570.192296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.294491] Lustre: 171325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2570.375324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2570.385295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2572.360394] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2576.922461] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2578.809351] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2578.924866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2580.635900] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2580.707863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2586.086055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2586.087968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2586.091237] LustreError: Skipped 1 previous similar message [ 2586.098696] Lustre: Skipped 1 previous similar message [ 2586.100210] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2586.107032] Lustre: Skipped 7 previous similar messages [ 2587.480119] Lustre: server umount lustre-OST0000 complete [ 2595.057844] Lustre: server umount lustre-MDT0000 complete [ 2596.322520] LustreError: 169367:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.334909] LustreError: 169367:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 41 previous similar messages [ 2596.760475] LustreError: 171324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784949906 with bad export cookie 6072905225282144454 [ 2596.763929] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.764644] LustreError: 171324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2596.888766] Lustre: server umount lustre-MDT0001 complete [ 2600.657186] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:25:09 (1784949909) [ 2603.548856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.561357] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2609.665597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.658655] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2614.569395] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2616.671317] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2619.576064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2622.069982] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2624.858811] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2634.281469] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:25:43 (1784949943) [ 2636.258284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2636.264527] LustreError: Skipped 1 previous similar message [ 2641.565714] Lustre: server umount lustre-OST0000 complete [ 2649.304950] Lustre: server umount lustre-MDT0000 complete [ 2650.677348] LustreError: 173717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784949960 with bad export cookie 6072905225282145336 [ 2650.681240] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2650.683830] LustreError: 173717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2650.789119] Lustre: server umount lustre-MDT0001 complete [ 2654.765601] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2655.979791] Key type lgssc unregistered [ 2656.147635] LNet: 177460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.152421] LNetError: 177460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.165566] LNet: Removed LNI 192.168.204.124@tcp [ 2656.601768] Key type .llcrypt unregistered [ 2656.603252] Key type ._llcrypt unregistered [ 2665.268517] Key type ._llcrypt registered [ 2665.269833] Key type .llcrypt registered [ 2665.335721] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2665.829238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2665.839411] alg: No test for adler32 (adler32-zlib) [ 2666.723157] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2666.843265] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2668.440156] Key type lgssc registered [ 2668.992193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2672.880416] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2672.892589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2688.352578] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2693.472357] LustreError: 178679:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.124@tcp: failed processing log, type 4: rc = -110 [ 2698.592489] LustreError: 178679:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2698.599950] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2698.603603] LustreError: MGC192.168.204.124@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2698.610243] LustreError: 178679:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2698.615479] LustreError: 178679:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 2698.619917] LustreError: 178679:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2698.642909] Lustre: server umount lustre-OST0000 complete [ 2698.645586] LustreError: 178679:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2701.554751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.716112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.144336] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2707.529657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.683534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2709.358330] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2712.489240] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2714.673034] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2717.329407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2717.442424] LustreError: 180991:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2717.447104] Lustre: 180991:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.124@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2717.521526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.106135] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2723.109936] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2728.418393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2728.424215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2728.434204] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2728.934638] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2728.936126] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2728.946347] Lustre: Skipped 1 previous similar message [ 2730.342488] Lustre: server umount lustre-MDT0000 complete [ 2732.044280] LustreError: 179159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950041 with bad export cookie 3470051489603950758 [ 2732.048814] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2732.050970] LustreError: 179159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2732.194261] Lustre: server umount lustre-MDT0001 complete [ 2743.993498] Lustre: server umount lustre-OST0000 complete [ 2747.361493] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 2751.796434] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2767.193537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.277890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2767.289831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2767.330860] Lustre: lustre-MDT0000: new disk, initializing [ 2767.351825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.358403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2768.934424] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2774.044384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.084192] Lustre: 184512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2774.106503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2774.108287] Lustre: Skipped 1 previous similar message [ 2774.140725] Lustre: lustre-MDT0001: new disk, initializing [ 2774.172720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2774.184248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2774.187637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2775.502365] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2779.841852] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2782.327316] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2785.087564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2785.242453] Lustre: lustre-OST0000: new disk, initializing [ 2785.246906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2785.251672] Lustre: 186477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2785.303192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.633080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2786.638595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2786.689062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2787.763883] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2792.525148] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2794.292066] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2794.391856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2796.262006] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2796.392317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2802.148768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2802.149020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2802.153858] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2802.160737] Lustre: Skipped 1 previous similar message [ 2802.164427] Lustre: Skipped 2 previous similar messages [ 2803.356767] Lustre: server umount lustre-OST0000 complete [ 2805.217795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2805.220546] LustreError: Skipped 1 previous similar message [ 2805.222784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2805.231101] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2812.385985] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2812.390318] Lustre: Skipped 3 previous similar messages [ 2819.040098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2819.116546] Lustre: server umount lustre-MDT0000 complete [ 2820.806607] LustreError: 184503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950130 with bad export cookie 3470051489603951458 [ 2820.812168] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2820.814523] LustreError: 184503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2820.958360] Lustre: server umount lustre-MDT0001 complete [ 2824.869152] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:28:53 (1784950133) [ 2831.845458] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2833.036940] Key type lgssc unregistered [ 2833.204490] LNet: 189419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2833.208877] LNetError: 189419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2833.217556] LNet: Removed LNI 192.168.204.124@tcp [ 2833.671107] Key type .llcrypt unregistered [ 2833.673041] Key type ._llcrypt unregistered [ 2841.719351] Key type ._llcrypt registered [ 2841.720534] Key type .llcrypt registered [ 2841.759724] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2842.077342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2842.089779] alg: No test for adler32 (adler32-zlib) [ 2842.931746] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2843.010882] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2844.584126] Key type lgssc registered [ 2844.877601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2847.151200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2847.157313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.294145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2849.526784] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2852.180156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.302449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2854.154625] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2857.060932] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2859.697263] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2862.980078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.103344] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2863.105979] LustreError: 192476:0:(llog.c:1696:llog_backup()) MGC192.168.204.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2863.112389] Lustre: 192476:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.124@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2863.177112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.776768] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2869.175707] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2870.157491] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2870.159797] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2870.165065] Lustre: Skipped 2 previous similar messages [ 2870.168223] Lustre: Skipped 1 previous similar message [ 2875.362755] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2875.369068] Lustre: Skipped 3 previous similar messages [ 2876.327380] Lustre: server umount lustre-MDT0000 complete [ 2877.414209] LustreError: 190631:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.421785] LustreError: 190631:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 2877.758093] LustreError: 190593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950187 with bad export cookie 6462097982244661560 [ 2877.766516] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.766522] LustreError: 190593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2879.456434] LustreError: 191341:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000015407ed9 x1871655890604288/t0(0) o900->lustre-MDT0000-lwp-MDT0001@192.168.204.124@tcp:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2879.535727] Lustre: server umount lustre-MDT0001 complete [ 2891.295886] Lustre: server umount lustre-OST0000 complete [ 2894.991331] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 2898.891398] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2914.714848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.838262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2914.858485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2914.915485] Lustre: lustre-MDT0000: new disk, initializing [ 2914.942067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.952621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2916.676494] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2922.203760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.261324] Lustre: 195995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2922.294285] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2922.296954] Lustre: Skipped 1 previous similar message [ 2922.299710] LustreError: 196778:0:(llog.c:1696:llog_backup()) MGC192.168.204.124@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2922.305778] LustreError: 196778:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 2922.310077] Lustre: 196778:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.124@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2922.323615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2922.327956] Lustre: Skipped 1 previous similar message [ 2922.380252] Lustre: lustre-MDT0001: new disk, initializing [ 2922.427823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.447875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2922.454992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2924.238966] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2928.935194] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2931.540641] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2934.270647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2934.408163] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2934.410180] Lustre: Skipped 1 previous similar message [ 2934.412258] LustreError: 197941:0:(llog.c:1696:llog_backup()) MGC192.168.204.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2934.417131] LustreError: 197941:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 2934.420613] Lustre: 197941:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.124@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2934.425390] Lustre: 197941:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2934.435919] Lustre: lustre-OST0000: new disk, initializing [ 2934.438112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2934.441229] Lustre: 197964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2934.478973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.951140] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2941.475949] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2941.963247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2941.970640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2942.011679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2943.189223] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2943.316168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2945.335514] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2945.442853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2947.053365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 2947.055274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2947.061119] LustreError: Skipped 1 previous similar message [ 2947.071863] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2947.078649] Lustre: Skipped 1 previous similar message [ 2948.453880] Lustre: server umount lustre-OST0000 complete [ 2952.162332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2952.163047] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 2952.166637] Lustre: Skipped 2 previous similar messages [ 2956.172481] Lustre: server umount lustre-MDT0000 complete [ 2957.281266] LustreError: 196796:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.286473] LustreError: 196796:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2957.526514] LustreError: 195988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950266 with bad export cookie 6462097982244662260 [ 2957.530047] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2957.530464] LustreError: 195988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2957.642721] Lustre: server umount lustre-MDT0001 complete [ 2960.600363] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:31:09 (1784950269) [ 2967.983344] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2972.224347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.379457] LustreError: 201084:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.392202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.992762] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2977.735818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.762673] LustreError: 201085:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.704839] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2981.155625] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2984.420415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.932469] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2990.950733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2991.062193] Lustre: lustre-OST0001: new disk, initializing [ 2991.066239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2991.072064] Lustre: 203393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2993.216856] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2994.141336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2994.148120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2994.196602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2999.282052] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3000.942651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3001.474475] Lustre: DEBUG MARKER: ADD OST3 [ 3003.663517] loop: module loaded [ 3003.697359] loop0: detected capacity change from 0 to 204800000 [ 3003.779013] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3003.782914] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3003.786557] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3006.190230] loop0: detected capacity change from 0 to 204800000 [ 3008.109071] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.215090] Lustre: lustre-OST0003: new disk, initializing [ 3008.218358] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3008.223115] Lustre: 205123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3008.285391] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3008.288859] Lustre: Skipped 3 previous similar messages [ 3008.886435] Lustre: DEBUG MARKER: STOP OST3 [ 3009.637381] Lustre: Failing over lustre-OST0003 [ 3009.698168] LustreError: 202570:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.708344] Lustre: server umount lustre-OST0003 complete [ 3010.562427] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3014.436925] Lustre: DEBUG MARKER: START OST3 again [ 3016.890745] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.568230] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3022.324947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3022.331325] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3022.344854] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3033.175102] Lustre: Failing over lustre-OST0003 [ 3033.214646] Lustre: server umount lustre-OST0003 complete [ 3037.676546] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 3037.676783] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3037.680506] LustreError: 202570:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.680514] LustreError: 202570:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 3037.681828] LustreError: Skipped 1 previous similar message [ 3037.690485] Lustre: Skipped 1 previous similar message [ 3047.909198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3047.911787] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 3047.913891] Lustre: Skipped 3 previous similar messages [ 3047.916219] Lustre: Skipped 3 previous similar messages [ 3052.159225] Lustre: server umount lustre-MDT0000 complete [ 3053.553131] LustreError: 201067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950362 with bad export cookie 6462097982244663128 [ 3053.559186] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3053.560578] LustreError: 201067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3058.146385] LustreError: 202287:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.147785] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3058.156260] LustreError: 202287:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 10 previous similar messages [ 3058.165703] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 3058.174959] Lustre: Skipped 1 previous similar message [ 3059.794238] Lustre: server umount lustre-MDT0001 complete [ 3061.355618] Lustre: server umount lustre-OST0000 complete [ 3062.985341] Lustre: server umount lustre-OST0001 complete [ 3065.305513] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 3068.237350] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3081.696932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.776560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3081.786918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3081.821414] Lustre: lustre-MDT0000: new disk, initializing [ 3081.841023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.843274] Lustre: Skipped 1 previous similar message [ 3081.848177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3083.088156] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3087.566787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.608233] Lustre: 210529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3087.702941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3089.061002] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3092.785314] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3094.531601] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3096.782946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.890574] Lustre: 212501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3098.690985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3098.694038] Lustre: Skipped 1 previous similar message [ 3098.696261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3098.731513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3098.765515] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3102.444828] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3104.312096] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3104.440351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3106.372363] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3106.503043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3108.839621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 3108.840948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3108.842257] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 3108.858947] Lustre: Skipped 2 previous similar messages [ 3113.571932] Lustre: server umount lustre-OST0000 complete [ 3118.562086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 3121.340382] Lustre: server umount lustre-MDT0000 complete [ 3123.045901] LustreError: 212500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950432 with bad export cookie 6462097982244665389 [ 3123.052911] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3123.056580] LustreError: 212500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3123.217350] Lustre: server umount lustre-MDT0001 complete [ 3124.806073] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:33:53 (1784950433) [ 3131.897941] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 3136.512310] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3151.542926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.618834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3151.630134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3151.632029] Lustre: Skipped 3 previous similar messages [ 3151.669349] Lustre: lustre-MDT0000: new disk, initializing [ 3151.670738] Lustre: Skipped 2 previous similar messages [ 3151.693905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3152.963314] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3157.787960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.866640] Lustre: 217101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3158.012600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3159.409404] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3163.858313] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3165.932510] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3168.593419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3168.728452] Lustre: 219070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3170.206132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3170.726015] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3174.870039] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3176.403113] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3176.514308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3178.159932] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3178.260895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3180.520893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3180.522158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 3180.523400] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 3180.523406] Lustre: Skipped 4 previous similar messages [ 3180.527722] Lustre: Skipped 3 previous similar messages [ 3180.532915] LustreError: Skipped 1 previous similar message [ 3185.323940] Lustre: server umount lustre-OST0000 complete [ 3188.706376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 3193.325902] Lustre: server umount lustre-MDT0000 complete [ 3194.742585] LustreError: 217895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950504 with bad export cookie 6462097982244666271 [ 3194.746622] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3194.748953] LustreError: 217895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3194.901233] Lustre: server umount lustre-MDT0001 complete [ 3198.783671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.980376] LustreError: 221283:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.662191] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3203.543438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.264225] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3208.516791] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3210.839725] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3214.183429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.371644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3214.376327] Lustre: Skipped 7 previous similar messages [ 3216.784259] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3220.053760] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3252.255769] Lustre: DEBUG MARKER: conf-sanity test_153a: @@@@@@ FAIL: check_mount failed [ 3278.507947] Lustre: DEBUG MARKER: == conf-sanity test 153b: added IPv6 NID support ========= 23:36:27 (1784950587) [ 3290.593755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 3290.598434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3290.604492] Lustre: Skipped 2 previous similar messages [ 3290.607254] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 3290.611098] Lustre: Skipped 4 previous similar messages [ 3296.356776] Lustre: server umount lustre-MDT0000 complete [ 3297.912435] LustreError: 221265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950607 with bad export cookie 6462097982244667153 [ 3297.918284] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.921883] LustreError: 221265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3298.273644] LustreError: 223123:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.282874] LustreError: 223123:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3304.137356] Lustre: server umount lustre-MDT0001 complete [ 3320.288098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3320.380249] Lustre: server umount lustre-OST0000 complete [ 3323.955067] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 3327.625120] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3343.616180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.766664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3343.782191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3343.785800] Lustre: Skipped 3 previous similar messages [ 3343.835821] Lustre: lustre-MDT0000: new disk, initializing [ 3343.837105] Lustre: Skipped 2 previous similar messages [ 3343.867150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3343.870878] Lustre: Skipped 2 previous similar messages [ 3345.366381] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3350.717096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.784638] Lustre: 227033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3350.958721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3350.966141] Lustre: Skipped 1 previous similar message [ 3352.656352] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3357.183122] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3359.495921] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3362.766378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.928688] Lustre: 229003:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3364.618753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3365.525575] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3369.954924] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3371.764662] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3371.877670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3373.666218] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3373.785342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3375.081110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 3375.087208] LustreError: Skipped 1 previous similar message [ 3380.773867] Lustre: server umount lustre-OST0000 complete [ 3388.394388] Lustre: server umount lustre-MDT0000 complete [ 3389.732449] LustreError: 229002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950699 with bad export cookie 6462097982244667944 [ 3389.736476] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.736728] LustreError: 229002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3389.870895] Lustre: server umount lustre-MDT0001 complete [ 3396.582371] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3400.294900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.849264] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3405.047130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3406.589561] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3407.632253] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3410.234079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3412.493452] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3415.828799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3415.911798] Lustre: lustre-OST0001: new disk, initializing [ 3415.913449] Lustre: Skipped 2 previous similar messages [ 3415.915132] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3415.918053] Lustre: Skipped 3 previous similar messages [ 3415.921611] Lustre: 234237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3417.034262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3417.039084] Lustre: Skipped 1 previous similar message [ 3417.065964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3417.990043] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3422.899314] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3558.103513] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:41:06 (1784950866) [ 3560.431380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3560.436462] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 3560.441306] Lustre: Skipped 9 previous similar messages [ 3560.446372] Lustre: Skipped 12 previous similar messages [ 3561.225742] Lustre: server umount lustre-MDT0000 complete [ 3563.124155] LustreError: 231912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950872 with bad export cookie 6462097982244668826 [ 3563.132417] LustreError: 231912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3563.133593] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.288976] Lustre: server umount lustre-MDT0001 complete [ 3574.732185] Lustre: server umount lustre-OST0000 complete [ 3586.620433] Lustre: server umount lustre-OST0001 complete [ 3589.466223] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 3593.721441] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3610.633153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3610.742923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3610.750721] Lustre: Skipped 1 previous similar message [ 3610.762498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3610.808066] Lustre: lustre-MDT0000: new disk, initializing [ 3610.837207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.842412] Lustre: Skipped 7 previous similar messages [ 3610.852392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3610.859716] Lustre: Skipped 3 previous similar messages [ 3612.736971] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3618.328229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.402579] Lustre: 238154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3618.541361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3619.928331] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3624.575810] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3627.172364] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3630.251702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.417735] Lustre: 240123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3632.063756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3632.971733] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3637.591972] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3639.355117] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3639.436482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3641.165970] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3641.262623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3642.347534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 3642.353910] LustreError: Skipped 1 previous similar message [ 3648.236854] Lustre: server umount lustre-OST0000 complete [ 3664.352132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3664.451531] Lustre: server umount lustre-MDT0000 complete [ 3666.306390] LustreError: 238146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784950975 with bad export cookie 6462097982244670814 [ 3666.311359] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3666.317145] LustreError: 238146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3666.474711] Lustre: server umount lustre-MDT0001 complete [ 3670.041696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.215618] LustreError: 242339:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3670.221489] LustreError: 242339:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 3671.935823] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3675.758171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.656975] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3680.537295] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3682.981761] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3685.880609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3688.671501] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3691.995439] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3722.965934] Lustre: DEBUG MARKER: conf-sanity test_153c: @@@@@@ FAIL: too few connection attempts [ 3947.585163] Lustre: DEBUG MARKER: == conf-sanity test 154: expand .. on rename after MDT backup restore ========================================================== 23:47:36 (1784951256) [ 3951.588192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3951.589561] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 3951.594519] Lustre: Skipped 7 previous similar messages [ 3951.596544] Lustre: Skipped 12 previous similar messages [ 3954.483524] Lustre: server umount lustre-MDT0000 complete [ 3956.006719] LustreError: 242321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784951265 with bad export cookie 6462097982244671696 [ 3956.012025] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3956.012159] LustreError: 242321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3956.150307] Lustre: server umount lustre-MDT0001 complete [ 3967.932157] Lustre: server umount lustre-OST0000 complete [ 3971.999056] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 3976.299909] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3993.030607] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3997.120709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.245455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3997.264317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3997.268348] Lustre: Skipped 3 previous similar messages [ 3997.313514] Lustre: lustre-MDT0000: new disk, initializing [ 3997.314947] Lustre: Skipped 2 previous similar messages [ 3997.342278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3997.345750] Lustre: Skipped 2 previous similar messages [ 3998.918731] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4003.842418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.893749] Lustre: 248718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4004.048881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4004.053048] Lustre: Skipped 1 previous similar message [ 4005.640234] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4011.054470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.194813] Lustre: 250352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4011.740402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4013.594045] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4018.295615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4020.073934] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4020.747302] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4025.712392] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4035.042275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 4037.839634] Lustre: server umount lustre-MDT0000 complete [ 4039.734615] Lustre: server umount lustre-MDT0001 complete [ 4050.774300] Lustre: server umount lustre-OST0000 complete [ 4061.821197] Lustre: server umount lustre-OST0001 complete [ 4068.379331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4072.649367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4076.255379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4080.465220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4087.162726] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4090.635428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.646068] Lustre: lustre-MDT0000: reset Object Index mappings [ 4090.802242] LustreError: 256159:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.807960] LustreError: 256159:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 4092.067084] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4094.774683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.780626] Lustre: lustre-MDT0001: reset Object Index mappings [ 4096.222036] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4097.165635] Lustre: 257301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4099.217918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4101.374322] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4101.418644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4104.132781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4105.262395] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4106.005076] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4109.131626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4117.661432] Lustre: server umount lustre-MDT0000 complete [ 4119.270173] Lustre: server umount lustre-MDT0001 complete [ 4131.086005] Lustre: server umount lustre-OST0000 complete [ 4143.003496] Lustre: server umount lustre-OST0001 complete [ 4148.907686] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:50:57 (1784951457) [ 4155.409795] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 4159.018242] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4174.496701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.602518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4174.608033] Lustre: Skipped 2 previous similar messages [ 4174.675241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.677372] Lustre: Skipped 13 previous similar messages [ 4176.046576] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4180.784224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.857654] Lustre: 263422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4180.867956] Lustre: 263422:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 4182.801618] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4187.354829] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4189.610717] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4192.527090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4192.629199] Lustre: 265391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4192.632582] Lustre: 265391:0:(osd_compat.c:1352:osd_object_spec_find()) Skipped 1 previous similar message [ 4194.273243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4194.552296] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4198.434508] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4199.912398] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4200.000390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4201.371658] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4201.440052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4208.225640] Lustre: server umount lustre-OST0000 complete [ 4215.832889] Lustre: server umount lustre-MDT0000 complete [ 4216.977045] LustreError: 265390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784951526 with bad export cookie 6462097982244681132 [ 4216.979854] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4216.980497] LustreError: 265390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4216.983981] LustreError: Skipped 2 previous similar messages [ 4217.107142] Lustre: server umount lustre-MDT0001 complete [ 4223.434090] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4227.361792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4229.240628] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4233.075185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.909560] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4238.641794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4241.051556] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4244.572611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4246.254383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4246.763283] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4251.737546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4255.607382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4255.615092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4264.203746] Lustre: server umount lustre-MDT0000 complete [ 4265.991387] Lustre: server umount lustre-MDT0001 complete [ 4277.208350] Lustre: server umount lustre-OST0000 complete [ 4289.342497] Lustre: server umount lustre-OST0001 complete [ 4296.716084] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4301.526580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.660425] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4307.936682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.971653] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4311.454084] Lustre: 274814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4314.744489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4317.538375] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4321.272095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4323.801195] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4327.918605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4327.926628] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4332.604821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4336.810284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4336.816356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4341.077152] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:54:09 (1784951649) [ 4354.974571] Lustre: server umount lustre-MDT0000 complete [ 4362.498257] Lustre: server umount lustre-MDT0001 complete [ 4370.117749] Lustre: server umount lustre-OST0000 complete [ 4386.272252] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4386.380981] Lustre: server umount lustre-OST0001 complete [ 4389.504841] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 4393.856179] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4415.529921] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4420.222557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.281683] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4428.111572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4430.052753] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4432.824586] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4432.827392] Lustre: Skipped 3 previous similar messages [ 4436.049704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4437.781513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4438.721897] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4444.232725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.790148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4447.213371] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4452.822258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4460.660099] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 23:56:09 (1784951769) [ 4464.097293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 4464.101871] LustreError: Skipped 5 previous similar messages [ 4464.103848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4464.110571] Lustre: Skipped 22 previous similar messages [ 4464.112712] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4464.116279] Lustre: Skipped 27 previous similar messages [ 4467.842962] Lustre: server umount lustre-MDT0000 complete [ 4469.903339] Lustre: server umount lustre-MDT0001 complete [ 4482.073467] Lustre: server umount lustre-OST0000 complete [ 4494.272480] Lustre: server umount lustre-OST0001 complete [ 4498.248982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.988207] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4503.922748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.847922] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4509.232303] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4511.708745] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4515.054509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4517.722826] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4521.117759] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4538.466880] loop1: detected capacity change from 0 to 204800000 [ 4538.554561] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4538.558587] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4538.563144] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4540.932065] loop1: detected capacity change from 0 to 204800000 [ 4543.486866] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.539320] LustreError: 285660:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2920: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4543.548521] LustreError: 285660:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2920 [ 4543.552636] LustreError: 285660:0:(mgs_handler.c:517:mgs_target_reg()) fs2920-MDT0001: Error checking target index: rc = -13 [ 4543.558692] LustreError: fs2920-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4543.564326] LustreError: 289411:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4543.568744] LustreError: 289411:0:(tgt_mount.c:2129:server_put_super()) no obd fs2920-MDT0001 [ 4543.572602] LustreError: 289411:0:(tgt_mount.c:116:server_deregister_mount()) fs2920-MDT0001 not registered [ 4543.590273] Lustre: server umount fs2920-MDT0001 complete [ 4543.592450] LustreError: 289411:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4547.732536] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.807675] Lustre: srv-fs2920-MDT0001: No data found on store. Initialize space. [ 4547.811980] Lustre: Skipped 14 previous similar messages [ 4547.866220] Lustre: fs2920-MDT0001: new disk, initializing [ 4547.867705] Lustre: Skipped 11 previous similar messages [ 4547.907681] Lustre: srv-fs2920-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4547.912062] Lustre: 289877:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2920-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4547.916967] LustreError: 289877:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2920-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4549.605985] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4553.409267] Lustre: server umount fs2920-MDT0001 complete [ 4559.472152] loop1: detected capacity change from 0 to 204800000 [ 4561.967097] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.111847] Lustre: 291444:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2920-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4562.120335] LustreError: 291444:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2920-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4563.977499] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4565.816966] Lustre: server umount fs2920-MDT0001 complete [ 4576.341995] Lustre: server umount lustre-MDT0000 complete [ 4580.211289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4581.984279] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4585.453421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.204.124@tcp (at 192.168.204.124@tcp) [ 4585.808887] loop1: detected capacity change from 0 to 204800000 [ 4588.508338] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.679826] Lustre: 293885:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2920-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4588.685081] LustreError: 293885:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2920-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4590.290345] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4592.171746] Lustre: server umount fs2920-MDT0001 complete [ 4605.929911] LustreError: 287501:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.941946] LustreError: 287501:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 59 previous similar messages [ 4606.750699] Lustre: server umount lustre-OST0000 complete [ 4614.777866] Lustre: server umount lustre-MDT0000 complete [ 4616.339134] Lustre: server umount lustre-MDT0001 complete [ 4620.900977] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 4621.812672] Key type lgssc unregistered [ 4621.927355] LNet: 296226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4621.929898] LNetError: 296226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4621.938281] LNet: Removed LNI 192.168.204.124@tcp [ 4622.222076] Key type .llcrypt unregistered [ 4622.223438] Key type ._llcrypt unregistered [ 4631.400862] Key type ._llcrypt registered [ 4631.402630] Key type .llcrypt registered [ 4631.449618] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4631.895203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4631.959590] alg: No test for adler32 (adler32-zlib) [ 4632.831090] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4632.959284] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 4634.576188] Key type lgssc registered [ 4635.143702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4639.149562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4639.162140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.333691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.736425] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4645.657109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.783646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4647.060675] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4649.602899] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4651.448207] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4654.022292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.165459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.230305] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.238573] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 4656.535017] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4659.945462] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4661.736078] LustreError: 299304:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.806696] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.820237] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 4669.929098] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.944136] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 4674.200674] loop1: detected capacity change from 0 to 204800000 [ 4674.306301] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4674.312729] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4674.320003] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4675.053629] LustreError: 299600:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.064029] LustreError: 299600:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 4676.749988] loop1: detected capacity change from 0 to 204800000 [ 4679.494227] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4679.582454] Lustre: fs19029-OST0000: new disk, initializing [ 4679.586080] Lustre: srv-fs19029-OST0000: No data found on store. Initialize space. [ 4679.591118] Lustre: 301207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4679.631482] Lustre: fs19029-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.864412] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4685.292966] LustreError: 299973:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.306247] LustreError: 299973:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 4686.281019] Lustre: server umount fs19029-OST0000 complete [ 4691.722836] loop1: detected capacity change from 0 to 204800000 [ 4694.259678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.350932] Lustre: fs19029-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.001937] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4701.305380] loop2: detected capacity change from 0 to 204800000 [ 4701.411190] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4701.414693] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4701.419337] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4701.439428] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4701.444909] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 9 previous similar messages [ 4703.961134] loop2: detected capacity change from 0 to 204800000 [ 4706.682585] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4706.737648] LustreError: fs19029-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4706.745707] LustreError: fs19029-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4706.751124] LustreError: 304246:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4706.755522] LustreError: 304246:0:(tgt_mount.c:2129:server_put_super()) no obd fs19029-OST0001 [ 4706.759273] LustreError: 304246:0:(tgt_mount.c:116:server_deregister_mount()) fs19029-OST0001 not registered [ 4706.777766] Lustre: server umount fs19029-OST0001 complete [ 4706.780734] LustreError: 304246:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4711.883997] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4711.956701] Lustre: fs19029-OST0001: new disk, initializing [ 4711.959885] Lustre: srv-fs19029-OST0001: No data found on store. Initialize space. [ 4711.964418] Lustre: 304798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4712.002618] Lustre: fs19029-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4714.580351] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4718.763717] Lustre: server umount fs19029-OST0000 complete [ 4723.185512] Lustre: server umount fs19029-OST0001 complete [ 4729.093860] loop1: detected capacity change from 0 to 204800000 [ 4731.696701] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.803115] Lustre: fs19029-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4734.435758] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.442477] LustreError: 299303:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 19 previous similar messages [ 4734.476255] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4737.663638] loop2: detected capacity change from 0 to 204800000 [ 4740.275160] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4742.592515] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4744.685182] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 192.168.204.124@tcp failed: rc = -19 [ 4746.577991] Lustre: server umount fs19029-OST0000 complete [ 4749.798480] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 192.168.204.124@tcp failed: rc = -19 [ 4750.390216] Lustre: server umount fs19029-OST0001 complete [ 4753.889236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 4753.893290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4753.900112] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4754.919758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4754.919888] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4754.929267] Lustre: Skipped 1 previous similar message [ 4760.036318] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4765.156199] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4765.163042] Lustre: Skipped 2 previous similar messages [ 4768.224156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4768.298079] Lustre: server umount lustre-OST0000 complete [ 4770.275336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4770.276750] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4770.286655] Lustre: Skipped 1 previous similar message [ 4776.247761] Lustre: server umount lustre-MDT0000 complete [ 4778.064257] LustreError: 297457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784952087 with bad export cookie 7706840360752227450 [ 4778.070830] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4778.072867] LustreError: 297457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4778.215820] Lustre: server umount lustre-MDT0001 complete [ 4782.941609] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 4784.210720] Key type lgssc unregistered [ 4784.328311] LNet: 310725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4784.331180] LNetError: 310725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4784.340368] LNet: Removed LNI 192.168.204.124@tcp [ 4784.779148] Key type .llcrypt unregistered [ 4784.781603] Key type ._llcrypt unregistered [ 4798.351850] Key type ._llcrypt registered [ 4798.353189] Key type .llcrypt registered [ 4798.394478] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 4806.039439] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4806.372675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4806.380515] alg: No test for adler32 (adler32-zlib) [ 4807.231641] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4807.322358] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 4808.896203] Key type lgssc registered [ 4809.453802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4825.764562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4825.783563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.886136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4826.902604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4826.950512] Lustre: lustre-MDT0000: new disk, initializing [ 4826.980312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4826.991050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4828.707687] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4834.522355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.594738] Lustre: 314570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4834.643159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4834.646907] Lustre: Skipped 1 previous similar message [ 4834.691430] Lustre: lustre-MDT0001: new disk, initializing [ 4834.727896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4834.741734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4834.745105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4836.376583] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4841.347695] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4844.112422] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4847.396055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4847.543849] Lustre: lustre-OST0000: new disk, initializing [ 4847.547731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4847.553873] Lustre: 316538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4847.605921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.262839] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4853.241263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4853.245138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4853.272424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4854.133230] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4912.886657] Lustre: Mounted lustre-client [ 4913.370935] Lustre: Unmounted lustre-client [ 4918.248353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4918.249805] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4918.257672] Lustre: Skipped 1 previous similar message [ 4918.265706] Lustre: Skipped 1 previous similar message [ 4919.517186] Lustre: server umount lustre-OST0000 complete [ 4921.825432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 4921.829843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4921.836277] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4923.362225] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 4923.362783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4927.330771] Lustre: server umount lustre-MDT0000 complete [ 4929.007966] LustreError: 317957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784952238 with bad export cookie 17308724491225112843 [ 4929.014148] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4929.016205] LustreError: 317957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4929.181729] Lustre: server umount lustre-MDT0001 complete [ 4933.766244] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 4935.213827] Key type lgssc unregistered [ 4935.381697] LNet: 318709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4935.386733] LNetError: 318709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4986.598591] LNet: Removed LNI 192.168.204.124@tcp [ 4987.111718] Key type .llcrypt unregistered [ 4987.114287] Key type ._llcrypt unregistered [ 4995.186171] Key type ._llcrypt registered [ 4995.187526] Key type .llcrypt registered [ 4995.233928] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5004.190714] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5004.634781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5004.642984] alg: No test for adler32 (adler32-zlib) [ 5005.529828] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5005.638343] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 5007.240199] Key type lgssc registered [ 5007.797142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5023.095107] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:05:31 (1784952331) [ 5026.327379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5026.338228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.450378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5027.463275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5027.509591] Lustre: lustre-MDT0000: new disk, initializing [ 5027.537951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5027.545687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5029.412277] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5035.072164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.149789] Lustre: 321897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5035.195045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5035.199234] Lustre: Skipped 1 previous similar message [ 5035.250049] Lustre: lustre-MDT0001: new disk, initializing [ 5035.292149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5035.310780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5035.316195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5037.086992] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5041.901334] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5044.507800] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5047.844100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5048.017140] Lustre: lustre-OST0000: new disk, initializing [ 5048.020139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5048.025057] Lustre: 323866:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5048.073713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.689379] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5053.958886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5053.966459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5054.015164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5055.226758] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5064.168716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 5064.168876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5064.171831] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5064.175022] LustreError: Skipped 1 previous similar message [ 5064.184690] Lustre: Skipped 1 previous similar message [ 5068.235605] Lustre: server umount lustre-OST0000 complete [ 5071.330233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 5071.336121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5071.346539] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5071.354072] Lustre: Skipped 1 previous similar message [ 5074.402672] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5074.403595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5079.522653] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5079.528330] Lustre: Skipped 2 previous similar messages [ 5083.619377] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5083.624394] Lustre: Skipped 1 previous similar message [ 5084.128151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5084.213423] Lustre: server umount lustre-MDT0000 complete [ 5086.110258] LustreError: 324588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784952395 with bad export cookie 10563554686155664272 [ 5086.117261] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5086.117882] LustreError: 324588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5086.279989] Lustre: server umount lustre-MDT0001 complete [ 5091.464094] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5092.941817] Key type lgssc unregistered [ 5093.120632] LNet: 325774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5093.126320] LNetError: 325774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5093.135635] LNet: Removed LNI 192.168.204.124@tcp [ 5093.633481] Key type .llcrypt unregistered [ 5093.636122] Key type ._llcrypt unregistered [ 5103.778513] Key type ._llcrypt registered [ 5103.779556] Key type .llcrypt registered [ 5103.816645] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5104.113441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5104.127975] alg: No test for adler32 (adler32-zlib) [ 5105.020631] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5105.144062] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 5106.752179] Key type lgssc registered [ 5107.343816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5110.781748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5110.794139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.973447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5113.725682] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5117.618155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.770197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5119.588081] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5122.961400] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5125.653818] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5128.900412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.042946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5131.565961] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5134.909524] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5149.077819] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5149.810485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5153.768590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 5153.768801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5153.771614] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5153.789528] Lustre: Skipped 2 previous similar messages [ 5156.620784] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.24@tcp (stopping) [ 5156.628900] Lustre: Skipped 2 previous similar messages [ 5157.978533] Lustre: server umount lustre-MDT0000 complete [ 5158.883222] LustreError: 327022:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5158.893767] LustreError: 327022:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 5159.735674] LustreError: 327760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784952469 with bad export cookie 13661290077550396426 [ 5159.742507] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5159.747161] LustreError: 327760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5161.725869] LustreError: 327020:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.731034] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.24@tcp (stopping) [ 5164.002481] LustreError: 329465:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.003123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5164.022344] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 5166.849054] LustreError: 327022:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5166.861700] LustreError: 327022:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 5170.147670] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.124@tcp (stopping) [ 5170.153214] Lustre: Skipped 2 previous similar messages [ 5171.966481] LustreError: 327020:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5171.978401] LustreError: 327020:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 5174.240214] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5174.355645] Lustre: server umount lustre-MDT0001 complete [ 5176.120802] Lustre: server umount lustre-OST0000 complete [ 5180.233541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5182.054154] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5183.397449] Lustre: 331225:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.124@tcp [ 5184.484208] Lustre: server umount MGS complete [ 5188.366654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5188.506297] LustreError: 331794:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.124@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5188.527702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5189.712087] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5192.821641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.938228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5194.162773] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5196.731073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5198.528286] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5200.738880] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5201.251740] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 5202.917903] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5203.436528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5208.037813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 5208.040762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5208.043502] LustreError: Skipped 1 previous similar message [ 5208.044277] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5208.053379] Lustre: Skipped 1 previous similar message [ 5208.056229] Lustre: Skipped 1 previous similar message [ 5210.589766] Lustre: server umount lustre-OST0000 complete [ 5213.157186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5213.166609] Lustre: Skipped 1 previous similar message [ 5218.448272] Lustre: server umount lustre-MDT0000 complete [ 5219.584236] LustreError: 334477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784952528 with bad export cookie 13661290077550397252 [ 5219.587336] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5219.588221] LustreError: 334477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5219.702745] Lustre: server umount lustre-MDT0001 complete [ 5223.043026] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5223.949051] Key type lgssc unregistered [ 5224.060353] LNet: 335226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5224.063132] LNetError: 335226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5224.073269] LNet: Removed LNI 192.168.204.124@tcp [ 5224.350440] Key type .llcrypt unregistered [ 5224.351533] Key type ._llcrypt unregistered [ 5230.145635] Key type ._llcrypt registered [ 5230.146802] Key type .llcrypt registered [ 5230.183537] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5238.527390] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5238.835493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5238.858381] alg: No test for adler32 (adler32-zlib) [ 5239.708776] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5239.790822] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 5241.368173] Key type lgssc registered [ 5241.946946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5257.509724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5257.522213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.613502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5258.626010] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5258.668295] Lustre: lustre-MDT0000: new disk, initializing [ 5258.690571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5258.697221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5260.416804] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5265.689054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.752189] Lustre: 338360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5265.792319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5265.796679] Lustre: Skipped 1 previous similar message [ 5265.842021] Lustre: lustre-MDT0001: new disk, initializing [ 5265.884173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5265.900645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5265.906099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5267.686654] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5272.616530] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5275.201995] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5278.580645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5278.728553] Lustre: lustre-OST0000: new disk, initializing [ 5278.732137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5278.737404] Lustre: 340332:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5278.784107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.095541] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5285.174829] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5286.399272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5286.403059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5286.436686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5286.931496] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5287.011128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5288.566654] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5288.645400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5291.499130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 5291.499817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5291.502151] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5291.504547] LustreError: Skipped 1 previous similar message [ 5291.511258] Lustre: Skipped 1 previous similar message [ 5295.417405] Lustre: server umount lustre-OST0000 complete [ 5301.731334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5301.732233] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5301.736431] Lustre: Skipped 1 previous similar message [ 5301.739248] Lustre: Skipped 1 previous similar message [ 5301.742622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 5303.481311] Lustre: server umount lustre-MDT0000 complete [ 5304.839348] LustreError: 338353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784952614 with bad export cookie 1584222902391190570 [ 5304.842221] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5304.842408] LustreError: 338353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5304.968991] Lustre: server umount lustre-MDT0001 complete [ 5307.643932] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:10:16 (1784952616) [ 5312.560398] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5315.546773] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5326.550730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.648790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5326.658048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5326.696239] Lustre: lustre-MDT0000: new disk, initializing [ 5326.715828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.720908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5327.949301] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5332.054168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.091989] Lustre: 345083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5332.113889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5332.115731] Lustre: Skipped 1 previous similar message [ 5332.154398] Lustre: lustre-MDT0001: new disk, initializing [ 5332.184519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5332.196210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5332.199297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5333.502247] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5337.019613] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5338.679642] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5340.770129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.879826] Lustre: lustre-OST0000: new disk, initializing [ 5340.881339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5340.883596] Lustre: 347052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5340.910617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.650905] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5346.151348] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5347.320945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5347.323866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5347.355644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5362.660030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.204.124@tcp failed: rc = -107 [ 5362.660372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5362.661317] Lustre: lustre-OST0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5362.661321] Lustre: Skipped 1 previous similar message [ 5362.668596] Lustre: Skipped 1 previous similar message [ 5366.098132] Lustre: server umount lustre-OST0000 complete [ 5367.322289] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 192.168.204.124@tcp failed: rc = -107 [ 5367.325359] LustreError: Skipped 1 previous similar message [ 5367.327093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5367.333886] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5367.336138] Lustre: Skipped 1 previous similar message [ 5367.419593] Lustre: server umount lustre-MDT0000 complete [ 5368.647111] LustreError: 345075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784952678 with bad export cookie 1584222902391191452 [ 5368.650360] LustreError: 345075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5368.650924] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.834355] Lustre: server umount lustre-MDT0001 complete [ 5378.092667] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5378.992300] Key type lgssc unregistered [ 5379.100404] LNet: 349201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5379.102761] LNetError: 349201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5379.111271] LNet: Removed LNI 192.168.204.124@tcp [ 5379.387044] Key type .llcrypt unregistered [ 5379.388050] Key type ._llcrypt unregistered [ 5418.661506] Key type ._llcrypt registered [ 5418.662935] Key type .llcrypt registered [ 5418.719432] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5418.861533] Key type .llcrypt unregistered [ 5418.863928] Key type ._llcrypt unregistered [ 5451.829475] Key type ._llcrypt registered [ 5451.830776] Key type .llcrypt registered [ 5451.868069] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5451.973302] Key type .llcrypt unregistered [ 5451.974260] Key type ._llcrypt unregistered [ 5465.643161] Key type ._llcrypt registered [ 5465.644331] Key type .llcrypt registered [ 5465.702518] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5465.834564] Key type .llcrypt unregistered [ 5465.835851] Key type ._llcrypt unregistered [ 5479.877592] Key type ._llcrypt registered [ 5479.878711] Key type .llcrypt registered [ 5479.917762] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5480.041363] Key type .llcrypt unregistered [ 5480.043265] Key type ._llcrypt unregistered [ 5518.358879] Key type ._llcrypt registered [ 5518.360225] Key type .llcrypt registered [ 5518.409139] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5518.553326] Key type .llcrypt unregistered [ 5518.555442] Key type ._llcrypt unregistered [ 5532.331944] Key type ._llcrypt registered [ 5532.333067] Key type .llcrypt registered [ 5532.379528] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5532.518377] Key type .llcrypt unregistered [ 5532.519987] Key type ._llcrypt unregistered [ 5544.443302] Key type ._llcrypt registered [ 5544.444297] Key type .llcrypt registered [ 5544.486434] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5544.600391] Key type .llcrypt unregistered [ 5544.603354] Key type ._llcrypt unregistered [ 5555.411651] Key type ._llcrypt registered [ 5555.412677] Key type .llcrypt registered [ 5555.448070] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5555.540198] Key type .llcrypt unregistered [ 5555.541294] Key type ._llcrypt unregistered [ 5566.380358] Key type ._llcrypt registered [ 5566.381470] Key type .llcrypt registered [ 5566.422431] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5566.520280] Key type .llcrypt unregistered [ 5566.521747] Key type ._llcrypt unregistered [ 5577.296160] Key type ._llcrypt registered [ 5577.298183] Key type .llcrypt registered [ 5577.339058] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5577.455312] Key type .llcrypt unregistered [ 5577.457038] Key type ._llcrypt unregistered [ 5598.563519] Key type ._llcrypt registered [ 5598.564613] Key type .llcrypt registered [ 5598.610043] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5598.711252] Key type .llcrypt unregistered [ 5598.712400] Key type ._llcrypt unregistered [ 5609.739164] Key type ._llcrypt registered [ 5609.740214] Key type .llcrypt registered [ 5609.784754] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5609.893190] Key type .llcrypt unregistered [ 5609.894169] Key type ._llcrypt unregistered [ 5620.849485] Key type ._llcrypt registered [ 5620.850600] Key type .llcrypt registered [ 5620.882569] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5620.971205] Key type .llcrypt unregistered [ 5620.972346] Key type ._llcrypt unregistered [ 5638.206165] Key type ._llcrypt registered [ 5638.207385] Key type .llcrypt registered [ 5638.245019] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5644.203344] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5644.703777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5644.719492] alg: No test for adler32 (adler32-zlib) [ 5645.575425] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5645.667635] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 5647.248135] Key type lgssc registered [ 5647.618692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5660.393435] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5660.400129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.496618] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5661.508899] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5661.544872] Lustre: tst250fs-MDT0000: new disk, initializing [ 5661.568937] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.575429] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5663.355989] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5669.059428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.132718] Lustre: 369118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5669.173602] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5669.178268] Lustre: Skipped 1 previous similar message [ 5669.228562] Lustre: tst250fs-MDT0001: new disk, initializing [ 5669.279635] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5669.299902] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5669.307685] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5670.871323] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5673.242741] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5675.660270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5675.809470] Lustre: tst250fs-OST0000: new disk, initializing [ 5675.813291] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5675.818499] Lustre: 370747:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5675.876105] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5677.962066] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5677.965535] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5678.024778] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5678.121813] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5682.966988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5683.068505] Lustre: tst250fs-OST0001: new disk, initializing [ 5683.072493] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5683.078058] Lustre: 371620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5683.144786] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5683.802887] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5683.810145] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5683.871318] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5685.707976] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5689.005916] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5704.681881] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 192.168.204.124@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5704.685361] Lustre: tst250fs-MDT0000: Not available for connect from 192.168.204.124@tcp (stopping) [ 5704.685524] Lustre: Skipped 3 previous similar messages [ 5704.687603] Lustre: Skipped 2 previous similar messages [ 5707.697501] Lustre: server umount tst250fs-MDT0000 complete [ 5709.215172] LustreError: 369154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784953018 with bad export cookie 789169427382423439 [ 5709.221655] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5709.362458] Lustre: server umount tst250fs-MDT0001 complete [ 5721.435407] Lustre: server umount tst250fs-OST0000 complete [ 5733.253408] Lustre: server umount tst250fs-OST0001 complete [ 5741.953715] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5745.618872] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5759.489733] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 00:17:48 (1784953068) [ 5760.334376] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5766.969288] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5771.290795] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5785.714989] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5681 sec ========== 00:18:14 (1784953094) [ 5786.649887] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:18:15 (1784953095) === [ 5788.289967] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:18:16 (1784953096) === [ 5805.599621] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5807.103134] Key type lgssc unregistered [ 5807.280603] LNet: 380733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5807.285447] LNetError: 380733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5807.298669] LNet: Removed LNI 192.168.204.124@tcp [ 5807.833172] Key type .llcrypt unregistered [ 5807.835402] Key type ._llcrypt unregistered