[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487256822 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002316] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006579] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009003] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013061] *** VALIDATE selinux *** [ 0.021326] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026347] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.031472] *** VALIDATE proc *** [ 0.032259] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035340] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037091] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039043] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.042872] debug: unmapping init [mem 0xffffffffb8859000-0xffffffffb8860fff] [ 0.044177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045664] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050019] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.053319] rcu: Hierarchical SRCU implementation. [ 0.055526] smp: Bringing up secondary CPUs ... [ 0.056619] x86: Booting SMP configuration: [ 0.057037] .... node #0, CPUs: #1 #2 #3 [ 0.061350] smp: Brought up 1 node, 4 CPUs [ 0.063066] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.139014] node 0 deferred pages initialised in 73ms [ 0.141349] devtmpfs: initialized [ 0.142193] x86/mm: Memory block size: 128MB [ 0.144232] gcov: version magic: 0x41383552 [ 0.145551] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147253] pinctrl core: initialized pinctrl subsystem [ 0.148209] [ 0.148770] ************************************************************* [ 0.149014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150012] ** ** [ 0.151024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152014] ** ** [ 0.153011] ** This means that this kernel is built to expose internal ** [ 0.154015] ** IOMMU data structures, which may compromise security on ** [ 0.155018] ** your system. ** [ 0.156013] ** ** [ 0.157021] ** If you see this message and you are not debugging the ** [ 0.158022] ** kernel, report this immediately to your vendor! ** [ 0.159009] ** ** [ 0.160015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161013] ************************************************************* [ 0.162527] NET: Registered protocol family 16 [ 0.163450] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.164065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.165075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.166353] cpuidle: using governor menu [ 0.167700] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168535] PCI: Using configuration type 1 for base access [ 0.169144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.176145] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.177044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178246] cryptd: max_cpu_qlen set to 1000 [ 0.181254] ACPI: Added _OSI(Module Device) [ 0.182018] ACPI: Added _OSI(Processor Device) [ 0.184015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190127] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197392] ACPI: Interpreter enabled [ 0.199076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200014] ACPI: Using IOAPIC for interrupt routing [ 0.202177] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.215806] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.222018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231798] acpiphp: Slot [2] registered [ 0.233149] acpiphp: Slot [5] registered [ 0.235150] acpiphp: Slot [6] registered [ 0.237122] acpiphp: Slot [7] registered [ 0.238112] acpiphp: Slot [8] registered [ 0.240167] acpiphp: Slot [9] registered [ 0.241117] acpiphp: Slot [10] registered [ 0.243136] acpiphp: Slot [3] registered [ 0.245109] acpiphp: Slot [4] registered [ 0.246083] acpiphp: Slot [11] registered [ 0.247098] acpiphp: Slot [12] registered [ 0.249136] acpiphp: Slot [13] registered [ 0.251137] acpiphp: Slot [14] registered [ 0.253136] acpiphp: Slot [15] registered [ 0.255121] acpiphp: Slot [16] registered [ 0.256123] acpiphp: Slot [17] registered [ 0.258124] acpiphp: Slot [18] registered [ 0.259126] acpiphp: Slot [19] registered [ 0.261161] acpiphp: Slot [20] registered [ 0.263104] acpiphp: Slot [21] registered [ 0.265128] acpiphp: Slot [22] registered [ 0.266144] acpiphp: Slot [23] registered [ 0.268124] acpiphp: Slot [24] registered [ 0.270133] acpiphp: Slot [25] registered [ 0.272138] acpiphp: Slot [26] registered [ 0.273122] acpiphp: Slot [27] registered [ 0.275118] acpiphp: Slot [28] registered [ 0.277146] acpiphp: Slot [29] registered [ 0.278173] acpiphp: Slot [30] registered [ 0.280123] acpiphp: Slot [31] registered [ 0.282063] PCI host bridge to bus 0000:00 [ 0.283025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.298040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.300190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303262] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306246] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.316021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320444] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.323018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.325017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.329519] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.332611] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.335053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.338807] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.355019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363034] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.368772] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.413828] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.421026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.430027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.450019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.465028] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.471014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.481020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.503020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.516358] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.524015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.532018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.550017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.560111] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.567017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.596133] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.602020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.635604] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637473] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.640342] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.643502] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.645234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.650119] iommu: Default domain type: Passthrough [ 0.653454] SCSI subsystem initialized [ 0.654173] ACPI: bus type USB registered [ 0.655193] usbcore: registered new interface driver usbfs [ 0.658183] usbcore: registered new interface driver hub [ 0.661148] usbcore: registered new device driver usb [ 0.664246] pps_core: LinuxPPS API ver. 1 registered [ 0.667014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.670066] PTP clock support registered [ 0.673095] EDAC MC: Ver: 3.0.0 [ 0.675137] PCI: Using ACPI for IRQ routing [ 0.676581] NetLabel: Initializing [ 0.677014] NetLabel: domain hash size = 128 [ 0.678010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.681093] NetLabel: unlabeled traffic allowed by default [ 0.683151] vgaarb: loaded [ 0.685330] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.687015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.694402] clocksource: Switched to clocksource kvm-clock [ 0.798280] VFS: Disk quotas dquot_6.6.0 [ 0.799846] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802363] *** VALIDATE ramfs *** [ 0.803233] *** VALIDATE hugetlbfs *** [ 0.805185] pnp: PnP ACPI init [ 0.807706] pnp: PnP ACPI: found 6 devices [ 0.838264] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.840938] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.843147] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.845464] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.847556] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.850265] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.853420] NET: Registered protocol family 2 [ 0.856041] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.861347] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.865509] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.871774] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.876121] TCP: Hash tables configured (established 65536 bind 65536) [ 0.879268] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.883223] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.886312] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.889535] NET: Registered protocol family 1 [ 0.892331] RPC: Registered named UNIX socket transport module. [ 0.894603] RPC: Registered udp transport module. [ 0.896451] RPC: Registered tcp transport module. [ 0.898437] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.901386] NET: Registered protocol family 44 [ 0.903216] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.905515] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.908026] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.910618] PCI: CLS 0 bytes, default 64 [ 0.912448] Unpacking initramfs... [ 2.310892] debug: unmapping init [mem 0xffff94bebcc54000-0xffff94bebffbffff] [ 2.316207] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.319464] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.323785] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.832314] Initialise system trusted keyrings [ 2.834300] Key type blacklist registered [ 2.837239] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.846462] zbud: loaded [ 2.850664] *** VALIDATE nfs *** [ 2.851962] *** VALIDATE nfs4 *** [ 2.854079] pstore: using deflate compression [ 2.857230] Platform Keyring initialized [ 2.955677] NET: Registered protocol family 38 [ 2.957612] Key type asymmetric registered [ 2.958770] Asymmetric key parser 'x509' registered [ 2.960134] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.963175] io scheduler mq-deadline registered [ 2.965103] io scheduler kyber registered [ 2.966808] io scheduler bfq registered [ 2.968382] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.971430] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.974719] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.977772] ACPI: Power Button [PWRF] [ 2.982866] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.988845] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.002769] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.011689] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.030724] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.056952] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.084079] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.088754] Non-volatile memory driver v1.3 [ 3.090838] Linux agpgart interface v0.103 [ 3.120087] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.124413] vda: detected capacity change from 0 to 69914624 [ 3.141202] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.145071] vdb: detected capacity change from 0 to 1073741824 [ 3.161693] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.165209] vdc: detected capacity change from 0 to 2621440000 [ 3.184403] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.187672] vdd: detected capacity change from 0 to 2621440000 [ 3.209600] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.213278] vde: detected capacity change from 0 to 4294967296 [ 3.228772] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.231350] vdf: detected capacity change from 0 to 4294967296 [ 3.240719] libphy: Fixed MDIO Bus: probed [ 3.245264] usbcore: registered new interface driver usbserial_generic [ 3.247658] usbserial: USB Serial support registered for generic [ 3.249671] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.256385] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.258117] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.260415] mousedev: PS/2 mouse device common for all mice [ 3.263921] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.265540] rtc_cmos 00:05: RTC can wake from S4 [ 3.272319] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.272579] rtc_cmos 00:05: registered as rtc0 [ 3.279449] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.282780] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.285462] intel_pstate: CPU model not supported [ 3.288842] hid: raw HID events driver (C) Jiri Kosina [ 3.290699] usbcore: registered new interface driver usbhid [ 3.293156] usbhid: USB HID core driver [ 3.294848] drop_monitor: Initializing network drop monitor service [ 3.297272] Initializing XFRM netlink socket [ 3.298634] NET: Registered protocol family 10 [ 3.301672] Segment Routing with IPv6 [ 3.303132] NET: Registered protocol family 17 [ 3.304809] mpls_gso: MPLS GSO support [ 3.309433] RAS: Correctable Errors collector initialized. [ 3.312297] AVX version of gcm_enc/dec engaged. [ 3.313827] AES CTR mode by8 optimization enabled [ 3.399238] sched_clock: Marking stable (3399195950, 0)->(4388645953, -989450003) [ 3.402488] registered taskstats version 1 [ 3.404900] Loading compiled-in X.509 certificates [ 3.406777] zswap: loaded using pool lzo/zbud [ 3.435464] Key type big_key registered [ 3.448497] Key type encrypted registered [ 3.450466] ima: No TPM chip found, activating TPM-bypass! [ 3.453223] ima: Allocated hash algorithm: sha1 [ 3.454682] ima: No architecture policies found [ 3.455764] evm: Initialising EVM extended attributes: [ 3.457276] evm: security.selinux [ 3.458349] evm: security.ima [ 3.459672] evm: security.capability [ 3.461314] evm: HMAC attrs: 0x1 [ 3.464126] rtc_cmos 00:05: setting system clock to 2026-06-11 22:33:14 UTC (1781217194) [ 3.469452] debug: unmapping init [mem 0xffffffffb9803000-0xffffffffb99fffff] [ 3.472982] debug: unmapping init [mem 0xffffffffb8582000-0xffffffffb8858fff] [ 3.483200] Write protecting the kernel read-only data: 28672k [ 3.486712] debug: unmapping init [mem 0xffffffffb6c03000-0xffffffffb6dfffff] [ 3.490410] debug: unmapping init [mem 0xffffffffb7514000-0xffffffffb75fffff] [ 3.527081] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.536672] systemd[1]: Detected virtualization kvm. [ 3.538449] systemd[1]: Detected architecture x86-64. [ 3.539932] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.566531] systemd[1]: No hostname configured. [ 3.568403] systemd[1]: Set hostname to . [ 3.570651] random: systemd: uninitialized urandom read (16 bytes read) [ 3.572771] systemd[1]: Initializing machine ID from random generator. [ 3.606318] random: ln: uninitialized urandom read (6 bytes read) [ 3.716233] random: systemd: uninitialized urandom read (16 bytes read) [ 3.719816] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.727238] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.731568] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.451614] device-mapper: uevent: version 1.0.3 [ 4.453613] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.216916] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.263732] virtio_net virtio0 ens2: renamed from eth0 [ 5.325242] scsi host0: ata_piix [ 5.340114] scsi host1: ata_piix [ 5.342128] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.344527] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.765226] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.966140] random: crng init done [ 9.967487] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.399682] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.501225] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.764354] SELinux: Disabled at runtime. [ 11.825831] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.834724] systemd[1]: Detected virtualization kvm. [ 11.836824] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.304354] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.307349] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.312180] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.315645] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.318613] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.329399] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.333232] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ 12.462386] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.688276] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.995722] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.001785] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.153360] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.171703] EDAC sbridge: Ver: 1.1.2 [ 14.624950] Key type dns_resolver registered [ 14.964702] NFS: Registering the id_resolver key type [ 14.967099] Key type id_resolver registered [ 14.969089] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg423-server login: [ 43.849591] libcfs: loading out-of-tree module taints kernel. [ 43.903704] Key type ._llcrypt registered [ 43.906923] Key type .llcrypt registered [ 44.030336] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 62.092645] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 63.467587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.478491] alg: No test for adler32 (adler32-zlib) [ 64.974775] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 66.007181] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 67.823608] Key type lgssc registered [ 69.993961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.685544] hrtimer: interrupt took 10534783 ns [ 86.308670] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.311745] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 147.610608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.642930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.981710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 149.040792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 149.188954] Lustre: lustre-MDT0000: new disk, initializing [ 149.260347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.282661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 154.356973] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.682058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.780665] Lustre: 6503:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.832155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.836992] Lustre: Skipped 1 previous similar message [ 169.941610] Lustre: lustre-MDT0001: new disk, initializing [ 170.015788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.071148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.094106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.268700] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.987921] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.585957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.922434] Lustre: lustre-OST0000: new disk, initializing [ 191.927531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.932728] Lustre: 8406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.986785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.840979] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.771115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.794874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.900837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 214.570533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.707770] Lustre: lustre-OST0001: new disk, initializing [ 214.712921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 214.720739] Lustre: 9466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.794308] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.692411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.702052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.803635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 221.459954] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.803169] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.222289] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.974726] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.873048] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing yml_node [ 259.577271] Lustre: DEBUG MARKER: Client: 2.17.53.75 [ 262.550909] Lustre: DEBUG MARKER: MDS: 2.17.53.75 [ 265.598628] Lustre: DEBUG MARKER: OSS: 2.17.53.75 [ 267.642694] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 18:37:37 EDT 2026 [ 286.585682] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 288.760759] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 307.683729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.686368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.703498] Lustre: Skipped 3 previous similar messages [ 307.735398] Lustre: Skipped 3 previous similar messages [ 312.005266] Lustre: server umount lustre-MDT0000 complete [ 315.320187] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781217506 with bad export cookie 10647180240782171191 [ 315.331123] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.340428] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 317.920424] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.921935] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.942382] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 317.963104] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.927853] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.940195] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.950241] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 320.964458] Lustre: Skipped 2 previous similar messages [ 321.692249] Lustre: server umount lustre-MDT0001 complete [ 325.525821] Lustre: server umount lustre-OST0000 complete [ 329.263423] Lustre: server umount lustre-OST0001 complete [ 335.753541] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 345.440882] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 382.294691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.555374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 382.605316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 382.689153] Lustre: lustre-MDT0000: new disk, initializing [ 382.747887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.761449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 386.984841] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 397.154836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.288043] Lustre: 14981:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 397.348932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 397.353406] Lustre: Skipped 1 previous similar message [ 397.411302] Lustre: lustre-MDT0001: new disk, initializing [ 397.492869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 397.525334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 397.540565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.861414] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 408.300887] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 412.459440] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 419.407039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.629250] Lustre: lustre-OST0000: new disk, initializing [ 419.637302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.663189] Lustre: 16725:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 419.803213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.369065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 421.385498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 421.497415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 426.530430] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 434.426764] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 438.433930] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 438.674530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 442.076853] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 442.312832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.947116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.949910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.954755] LustreError: Skipped 1 previous similar message [ 446.956176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.971109] Lustre: Skipped 2 previous similar messages [ 446.980552] Lustre: Skipped 1 previous similar message [ 450.364096] Lustre: server umount lustre-OST0000 complete [ 457.185785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.189791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.193774] Lustre: Skipped 1 previous similar message [ 457.210360] Lustre: Skipped 1 previous similar message [ 461.142657] Lustre: server umount lustre-MDT0000 complete [ 462.306116] LustreError: 14987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.323252] LustreError: 14987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 464.959730] LustreError: 15762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781217655 with bad export cookie 10647180240782172724 [ 464.974439] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.980645] LustreError: 15762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 465.440971] Lustre: server umount lustre-MDT0001 complete [ 471.762826] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:41:01 (1781217661) [ 489.915441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.362372] LustreError: 19452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.488146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.476562] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 495.599077] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.904570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.587938] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 517.402225] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 522.123797] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 524.593119] Lustre: Failing over lustre-MDT0000 [ 524.872113] Lustre: server umount lustre-MDT0000 complete [ 525.796213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.796213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.796607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 525.820329] LustreError: 20660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.820340] LustreError: 20660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 541.153251] Lustre: 3650:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781217716/real 1781217716] req@ffff94be03423840 x1867741666633216/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781217732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 541.185644] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.718023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.274651] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.309938] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 551.394385] LustreError: 3649:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94be034221c0 x1867741666636032/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 557.024737] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 557.034470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 557.066413] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 557.335390] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 564.807984] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 571.363269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.369730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.377090] Lustre: Skipped 1 previous similar message [ 571.386376] Lustre: Skipped 1 previous similar message [ 575.051233] Lustre: server umount lustre-MDT0000 complete [ 578.565648] LustreError: 21775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781217769 with bad export cookie 10647180240782174138 [ 578.588134] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.588991] LustreError: 21775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 579.107904] Lustre: server umount lustre-MDT0001 complete [ 587.105654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.552264] LustreError: 22451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.570659] LustreError: 22451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 587.641106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.653843] Lustre: Skipped 2 previous similar messages [ 592.331609] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 602.046453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.193505] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 613.654855] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.986639] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 620.780975] Lustre: Failing over lustre-MDT0000 [ 621.182728] Lustre: server umount lustre-MDT0000 complete [ 623.073679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.074991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.456822] Lustre: 3651:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781217814/real 1781217814] req@ffff94bf1977a1c0 x1867741666661248/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781217830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 639.496729] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.880936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.035628] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 655.065363] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 655.332540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 655.371719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 655.379305] Lustre: Skipped 1 previous similar message [ 655.411420] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 664.534832] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 669.670409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 669.684024] Lustre: Skipped 1 previous similar message [ 674.823548] Lustre: server umount lustre-MDT0000 complete [ 677.856483] LustreError: 22451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.869509] LustreError: 22451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 678.355136] LustreError: 22431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781217869 with bad export cookie 10647180240782175076 [ 678.369749] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 678.371036] LustreError: 22431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 678.704052] Lustre: server umount lustre-MDT0001 complete [ 688.835772] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:44:38 (1781217878) [ 705.768562] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 716.232945] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 761.973022] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 773.409081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.802382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 773.854516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 774.002414] Lustre: lustre-MDT0000: new disk, initializing [ 774.119305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 774.127219] Lustre: Skipped 2 previous similar messages [ 774.174832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 778.958840] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 790.297920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.415665] Lustre: 28435:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 790.733565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 796.306738] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 806.071315] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 811.014891] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 818.320974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 818.492497] Lustre: 30179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 819.560534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 819.601889] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 819.717653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 819.740110] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 823.708194] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 831.406167] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 837.855954] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 865.764829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 865.765410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.783020] LustreError: Skipped 1 previous similar message [ 865.829718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 865.836955] Lustre: Skipped 4 previous similar messages [ 865.848953] Lustre: Skipped 3 previous similar messages [ 868.167995] Lustre: server umount lustre-OST0000 complete [ 872.932188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 879.091623] Lustre: server umount lustre-MDT0000 complete [ 881.123273] LustreError: 28442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.150519] LustreError: 28442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 883.253188] LustreError: 28427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218074 with bad export cookie 10647180240782175482 [ 883.255301] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 883.264997] LustreError: 28427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 883.773810] Lustre: server umount lustre-MDT0001 complete [ 896.152167] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 898.754172] Key type lgssc unregistered [ 899.139093] LNet: 32028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.151933] LNetError: 32028:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.166706] LNet: Removed LNI 192.168.204.123@tcp [ 900.013185] Key type .llcrypt unregistered [ 900.016309] Key type ._llcrypt unregistered [ 924.899549] Key type ._llcrypt registered [ 924.902350] Key type .llcrypt registered [ 925.035448] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 925.689489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.832573] alg: No test for adler32 (adler32-zlib) [ 927.372647] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 927.667579] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 929.327325] Key type lgssc registered [ 930.579474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 938.693597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 938.719207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.178783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.994774] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 950.763510] LustreError: 33424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.778996] LustreError: 33424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 954.595713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.878049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 960.266769] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 967.225239] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 971.616377] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 979.321267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 979.437984] LustreError: 35020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.691479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.078736] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 985.400831] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 990.260345] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 991.222991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1006.182850] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:49:55 (1781218195) [ 1008.483568] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1010.403856] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1018.429472] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:50:08 (1781218208) [ 1030.454893] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:50:19 (1781218219) [ 1031.689916] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1043.604866] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1043.618765] Lustre: Skipped 2 previous similar messages [ 1045.913986] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:50:35 (1781218235) [ 1047.907765] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1047.921239] Lustre: Skipped 2 previous similar messages [ 1055.942824] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1055.950714] Lustre: Skipped 1 previous similar message [ 1071.766961] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:51:01 (1781218261) [ 1074.740767] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1080.025373] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1089.381032] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:51:19 (1781218279) [ 1092.099732] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1092.103396] Lustre: Skipped 9 previous similar messages [ 1106.630317] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:51:36 (1781218296) [ 1123.297889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1123.305605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.318164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.322961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.327190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.327617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.351809] Lustre: server umount lustre-OST0000 complete [ 1134.048635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.057380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.075840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1134.080821] Lustre: Skipped 1 previous similar message [ 1139.680615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.684956] Lustre: Skipped 2 previous similar messages [ 1144.801689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.805546] Lustre: Skipped 1 previous similar message [ 1147.359171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1147.544972] Lustre: server umount lustre-MDT0000 complete [ 1149.923613] LustreError: 33424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.948737] LustreError: 33424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1151.028639] LustreError: 37511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218342 with bad export cookie 14653942256087680604 [ 1151.038366] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.042749] LustreError: 37511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1151.486611] Lustre: server umount lustre-MDT0001 complete [ 1162.915380] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1165.637863] Key type lgssc unregistered [ 1165.968214] LNet: 40937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.978268] LNetError: 40937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.992490] LNet: Removed LNI 192.168.204.123@tcp [ 1167.013751] Key type .llcrypt unregistered [ 1167.016230] Key type ._llcrypt unregistered [ 1184.719862] Key type ._llcrypt registered [ 1184.722133] Key type .llcrypt registered [ 1184.859028] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 1201.800390] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1203.243068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.289768] alg: No test for adler32 (adler32-zlib) [ 1204.460472] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 1204.725894] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1206.377091] Key type lgssc registered [ 1207.741684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1245.553627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1245.594595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.914703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1246.951760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1247.056924] Lustre: lustre-MDT0000: new disk, initializing [ 1247.150819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.165508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1251.312869] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1262.992456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.166881] Lustre: 44058:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1263.243873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1263.248399] Lustre: Skipped 1 previous similar message [ 1263.373745] Lustre: lustre-MDT0001: new disk, initializing [ 1263.481295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1263.541064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1263.554711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1268.278139] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1275.750348] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1280.053873] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1287.240770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1287.501902] Lustre: lustre-OST0000: new disk, initializing [ 1287.507469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1287.517958] Lustre: 45798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1287.659840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.392826] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1296.993302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1297.004215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1297.130826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1302.728401] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1307.285602] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1307.531344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1312.681240] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1312.930124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1317.861232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1317.866715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.875149] LustreError: Skipped 1 previous similar message [ 1317.883397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.898053] Lustre: Skipped 1 previous similar message [ 1321.275670] Lustre: server umount lustre-OST0000 complete [ 1328.096770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.106366] Lustre: Skipped 1 previous similar message [ 1328.111956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1328.116098] Lustre: Skipped 1 previous similar message [ 1331.579191] Lustre: server umount lustre-MDT0000 complete [ 1335.822828] LustreError: 44049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218526 with bad export cookie 17021997297603219021 [ 1335.844748] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1335.856933] LustreError: 44049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1336.133285] Lustre: server umount lustre-MDT0001 complete [ 1346.677975] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:55:36 (1781218536) [ 1353.629075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.148560] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.163746] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1354.216915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.165987] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1359.335583] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.450368] LustreError: 48113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.514734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.842211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1373.474138] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1379.521457] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1384.139788] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1392.116556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.602041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.903983] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1405.138750] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1409.917915] LustreError: 50301:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1411.394820] Lustre: Modifying parameter general.timeout=1 in log params [ 1412.722337] Lustre: Modifying parameter general.timeout=2 in log params [ 1414.969201] Lustre: Modifying parameter general.timeout=4 in log params [ 1414.973644] Lustre: Skipped 1 previous similar message [ 1419.177652] Lustre: Modifying parameter general.timeout=7 in log params [ 1419.186507] Lustre: Skipped 2 previous similar messages [ 1428.312572] Lustre: Modifying parameter general.timeout=15 in log params [ 1428.320030] Lustre: Skipped 7 previous similar messages [ 1445.213286] Lustre: Modifying parameter general.timeout=32 in log params [ 1445.219182] Lustre: Skipped 16 previous similar messages [ 1477.280776] Lustre: Modifying parameter general.timeout=68 in log params [ 1477.284689] Lustre: Skipped 35 previous similar messages [ 1545.788948] Lustre: Disabling parameter general.timeout= in log params [ 1545.792707] Lustre: Skipped 61 previous similar messages [ 1549.847156] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:58:59 (1781218739) [ 1552.484794] Lustre: 57251:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1554.423162] Lustre: 57349:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1558.521500] Lustre: 57561:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1558.542631] Lustre: 57561:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1591.068558] Lustre: 59122:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1591.095269] Lustre: 59122:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1591.208086] LustreError: 59122:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1596.007201] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:59:45 (1781218785) [ 1610.726706] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.735059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1610.746214] Lustre: Skipped 2 previous similar messages [ 1610.757128] Lustre: Skipped 2 previous similar messages [ 1615.848853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.857922] Lustre: Skipped 2 previous similar messages [ 1616.036409] Lustre: server umount lustre-MDT0000 complete [ 1620.024762] LustreError: 48094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218811 with bad export cookie 17021997297603219896 [ 1620.029871] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1620.038955] LustreError: 48094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1620.407571] Lustre: server umount lustre-MDT0001 complete [ 1634.810304] Lustre: server umount lustre-OST0000 complete [ 1661.159331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.437094] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1661.751840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.895762] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1675.249514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.379383] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1675.407654] Lustre: 61655:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1679.843561] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1685.019237] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1688.561893] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1695.224305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1695.411236] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1695.675525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.678699] Lustre: Skipped 1 previous similar message [ 1701.474318] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1706.671913] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1708.210964] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1708.216281] Lustre: Skipped 23 previous similar messages [ 1708.254934] Lustre: 63891:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1718.246727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1718.267066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.295833] Lustre: Skipped 1 previous similar message [ 1718.301937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.384389] Lustre: server umount lustre-OST0000 complete [ 1726.951494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1726.959807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.982425] Lustre: Skipped 1 previous similar message [ 1727.002636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1727.007151] Lustre: Skipped 1 previous similar message [ 1731.414408] Lustre: server umount lustre-MDT0000 complete [ 1734.121116] LustreError: 61666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.138866] LustreError: 61666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1735.148960] LustreError: 64352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218926 with bad export cookie 17021997297603223025 [ 1735.155216] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1735.158698] LustreError: 64352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1735.504856] Lustre: server umount lustre-MDT0001 complete [ 1747.839568] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1750.772990] Key type lgssc unregistered [ 1751.089450] LNet: 65086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.100959] LNetError: 65086:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.121175] LNet: Removed LNI 192.168.204.123@tcp [ 1752.055106] Key type .llcrypt unregistered [ 1752.057921] Key type ._llcrypt unregistered [ 1780.400540] Key type ._llcrypt registered [ 1780.402214] Key type .llcrypt registered [ 1780.684948] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1783.047818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1783.090798] alg: No test for adler32 (adler32-zlib) [ 1784.248213] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 1784.619355] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1786.322687] Key type lgssc registered [ 1787.250625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1795.626347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1795.639540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.129418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.424722] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1807.330609] LustreError: 66489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.358828] LustreError: 66489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1810.420349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.270924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1815.537414] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1820.845195] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1824.454828] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1831.093715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1832.083591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.025449] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1844.732421] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1853.304944] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1863.140760] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.143150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.162583] Lustre: Skipped 2 previous similar messages [ 1863.182521] Lustre: Skipped 2 previous similar messages [ 1868.258789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1868.272989] Lustre: Skipped 2 previous similar messages [ 1868.747742] Lustre: server umount lustre-MDT0000 complete [ 1872.410038] LustreError: 66469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219063 with bad export cookie 17798537881519961108 [ 1872.412958] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.434800] LustreError: 66469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1872.853684] Lustre: server umount lustre-MDT0001 complete [ 1886.358832] Lustre: server umount lustre-OST0000 complete [ 1917.888904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.356742] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1918.387366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1918.576798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.396495] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1933.486348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.838171] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1933.976187] Lustre: 71020:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1935.137492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1942.431746] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1949.863553] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1953.874810] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1962.503996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.640225] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1962.874848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.649699] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1974.020516] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1975.597892] Lustre: 73257:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1977.377190] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1981.922377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.948839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1981.949735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.791747] Lustre: server umount lustre-OST0000 complete [ 1991.138519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1991.153355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.172514] Lustre: Skipped 1 previous similar message [ 1991.194894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.211715] Lustre: Skipped 1 previous similar message [ 1997.202542] Lustre: server umount lustre-MDT0000 complete [ 1997.282614] LustreError: 71031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.327446] LustreError: 71031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2001.405857] LustreError: 71012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219192 with bad export cookie 17798537881519961941 [ 2001.417738] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.422719] LustreError: 71012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2002.045816] Lustre: server umount lustre-MDT0001 complete [ 2011.195753] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2013.720410] Key type lgssc unregistered [ 2014.000461] LNet: 74451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.010394] LNetError: 74451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.031796] LNet: Removed LNI 192.168.204.123@tcp [ 2014.817085] Key type .llcrypt unregistered [ 2014.819468] Key type ._llcrypt unregistered [ 2037.114676] Key type ._llcrypt registered [ 2037.119824] Key type .llcrypt registered [ 2037.225632] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2038.472115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2038.491249] alg: No test for adler32 (adler32-zlib) [ 2039.487440] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 2039.686254] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2041.343245] Key type lgssc registered [ 2042.427714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2049.811414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2049.839365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.546789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.258305] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2061.796456] LustreError: 75848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.817856] LustreError: 75848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2064.861680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.434955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.063205] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2076.145629] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2079.300831] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2084.993201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.669430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.693934] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2096.380123] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2105.795611] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2106.654263] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2108.118950] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2108.131545] Lustre: Skipped 1 previous similar message [ 2110.508085] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2110.512241] Lustre: Skipped 2 previous similar messages [ 2115.381046] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2115.387469] Lustre: Skipped 5 previous similar messages [ 2123.911278] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2123.915844] Lustre: Skipped 8 previous similar messages [ 2139.986836] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2140.003936] Lustre: Skipped 20 previous similar messages [ 2172.341949] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2172.356120] Lustre: Skipped 39 previous similar messages [ 2197.861040] Lustre: 83616:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2200.202809] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:09:50 (1781219390) [ 2201.642941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2203.501353] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:09:53 (1781219393) [ 2208.739219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2208.745321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.755653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2209.765927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2209.768181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.771610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2213.247480] Lustre: server umount lustre-OST0000 complete [ 2218.978320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2218.985821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.004757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2219.014765] Lustre: Skipped 1 previous similar message [ 2223.468231] Lustre: server umount lustre-MDT0000 complete [ 2225.123181] LustreError: 76562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.148316] LustreError: 76562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2227.300343] LustreError: 75828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219418 with bad export cookie 18056539894676555099 [ 2227.319645] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2227.710840] Lustre: server umount lustre-MDT0001 complete [ 2238.015865] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2241.084377] Key type lgssc unregistered [ 2241.433704] LNet: 84848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.442934] LNetError: 84848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2241.469832] LNet: Removed LNI 192.168.204.123@tcp [ 2242.470812] Key type .llcrypt unregistered [ 2242.473466] Key type ._llcrypt unregistered [ 2246.758380] Key type ._llcrypt registered [ 2246.760523] Key type .llcrypt registered [ 2246.884642] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2251.537727] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules [ 2252.727188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.768315] alg: No test for adler32 (adler32-zlib) [ 2253.815635] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 2254.090711] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2255.791208] Key type lgssc registered [ 2257.307891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2258.634161] LustreError: 86028:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2273.312795] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2279.944773] LustreError: 86028:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2283.147378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.168522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.816097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.092841] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2294.241809] LustreError: 87022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.272327] LustreError: 87022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2298.703398] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:11:28 (1781219488) [ 2299.363171] LustreError: 87023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.134908] LustreError: 87054:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2304.151885] LustreError: 87054:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2304.166454] LustreError: 87054:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2304.739644] Lustre: server umount lustre-MDT0000 complete [ 2320.379917] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2323.684286] Key type lgssc unregistered [ 2323.994387] LNet: 88514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.002948] LNetError: 88514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.017811] LNet: Removed LNI 192.168.204.123@tcp [ 2324.898534] Key type .llcrypt unregistered [ 2324.902619] Key type ._llcrypt unregistered [ 2344.569984] Key type ._llcrypt registered [ 2344.572231] Key type .llcrypt registered [ 2344.718653] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2345.997414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.162814] alg: No test for adler32 (adler32-zlib) [ 2347.263822] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 2347.458874] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2349.175313] Key type lgssc registered [ 2350.377515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2358.157925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2358.174406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.744661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.147519] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2369.011176] LustreError: 89716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.027066] LustreError: 89716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2372.934306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.612938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.896245] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2383.503205] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2387.371964] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2393.834514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2394.419932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.511320] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2406.502891] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2454.126193] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:14:03 (1781219643) [ 2455.630377] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2457.216515] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:14:07 (1781219647) [ 2465.760460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2465.776473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.805239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.286519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.333750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.352108] Lustre: Skipped 2 previous similar messages [ 2467.909154] Lustre: server umount lustre-MDT0000 complete [ 2471.401589] LustreError: 89716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.420282] LustreError: 89716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2471.895349] LustreError: 91342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219662 with bad export cookie 6907000083165943749 [ 2471.903631] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.906119] LustreError: 91342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2472.461659] Lustre: server umount lustre-MDT0001 complete [ 2486.756540] Lustre: server umount lustre-OST0000 complete [ 2498.612661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.154533] LustreError: 93669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.350528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.606433] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2504.688642] LustreError: 93670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.800628] LustreError: 93669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.840190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.753923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2519.042943] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2525.314336] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2529.886358] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2543.375188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2543.601622] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2543.613868] LustreError: 93659:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2543.648314] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2543.661352] LustreError: 95528:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2543.666540] LustreError: 95528:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2543.672535] LustreError: 95528:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2543.785484] Lustre: server umount lustre-OST0000 complete [ 2543.788958] LustreError: 95528:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2552.591413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2552.861531] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2552.867942] LustreError: 93658:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2552.888292] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2552.910203] LustreError: 95990:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2552.913208] LustreError: 95990:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2552.925122] LustreError: 95990:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2553.013482] Lustre: server umount lustre-OST0000 complete [ 2553.015220] LustreError: 95990:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2563.435952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2563.709405] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2563.715265] Lustre: Found index 0 for lustre-OST0000, updating log [ 2563.720360] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2563.931144] Lustre: lustre-OST0000: new disk, initializing [ 2563.950323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2563.969080] Lustre: 96524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2564.312823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.323624] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2575.106446] Lustre: Failing over lustre-OST0000 [ 2575.275172] Lustre: server umount lustre-OST0000 complete [ 2579.949270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2579.956616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.960551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2579.996661] Lustre: Skipped 2 previous similar messages [ 2583.078775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.095024] Lustre: Skipped 2 previous similar messages [ 2588.129036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.133977] Lustre: Skipped 1 previous similar message [ 2589.251832] Lustre: server umount lustre-MDT0000 complete [ 2593.221315] LustreError: 93652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219784 with bad export cookie 6907000083165946486 [ 2593.237328] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.718219] Lustre: server umount lustre-MDT0001 complete [ 2601.819530] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:16:31 (1781219791) [ 2618.191831] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2629.926409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.402836] LustreError: 98836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.506561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.087570] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2644.868670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.665256] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2653.447799] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2659.953720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2660.570088] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2666.872992] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2670.830639] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2670.849097] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2670.875775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2675.944892] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2675.973278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2675.985145] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2675.995153] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2676.031433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2676.401103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2676.517834] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2676.630706] Lustre: lustre-OST0001: new disk, initializing [ 2676.636896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2676.647692] Lustre: 101102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2676.973273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2676.985539] Lustre: Skipped 2 previous similar messages [ 2683.265585] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2687.076135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2687.084474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2687.197766] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2691.332830] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2698.678434] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2702.826586] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.852773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2702.868065] Lustre: Skipped 1 previous similar message [ 2702.894238] Lustre: Skipped 1 previous similar message [ 2707.431677] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2707.447688] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.080270] Lustre: server umount lustre-MDT0001 complete [ 2711.884737] LustreError: 98831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.939319] LustreError: 98831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2719.799488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.961505] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2719.971859] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2720.031405] Lustre: 98826:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2721.527675] LustreError: 102631:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2721.536525] LustreError: 102631:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2721.558754] LustreError: 98828:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2721.654179] LustreError: 102641:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.123@tcp: cfg command failed: rc = -17 [ 2721.660971] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2721.660971] [ 2725.392504] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2725.875426] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2735.623575] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:18:45 (1781219925) [ 2754.814440] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2773.494318] Lustre: 104667:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2796.354930] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2800.217966] Lustre: 105804:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2893.024217] Lustre: server umount lustre-MDT0000 complete [ 2894.305401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2894.313220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.323878] LustreError: 99550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.336767] LustreError: 99550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2897.043736] LustreError: 98816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220088 with bad export cookie 6907000083165947424 [ 2897.049450] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.060472] LustreError: 98816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2897.561480] Lustre: server umount lustre-MDT0001 complete [ 2912.997234] Lustre: server umount lustre-OST0000 complete [ 2915.873707] Lustre: 89026:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781220090/real 1781220090] req@ffff94be08c89e00 x1867744058625408/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781220106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2915.913216] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.938769] Lustre: Skipped 3 previous similar messages [ 2917.333755] Lustre: server umount lustre-OST0001 complete [ 2922.792189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2932.690641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2942.961519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2952.801556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2973.494455] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2985.850711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.866501] Lustre: lustre-MDT0000: reset Object Index mappings [ 2986.322141] LustreError: 109739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2986.329748] LustreError: 109739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2986.454423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.459180] Lustre: Skipped 1 previous similar message [ 2991.229890] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3000.114848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.128134] Lustre: lustre-MDT0001: reset Object Index mappings [ 3004.824970] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3007.715250] Lustre: 110852:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3013.606412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3020.072947] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3025.794986] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3028.141452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3031.819661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3031.828321] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3031.935670] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3034.678465] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3041.885335] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3045.928916] Lustre: 112698:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3101.326349] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:24:50 (1781220290) [ 3128.296405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3128.304885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.318421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.327880] Lustre: Skipped 4 previous similar messages [ 3131.451832] Lustre: server umount lustre-MDT0000 complete [ 3134.439205] LustreError: 109740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.462265] LustreError: 109740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3134.971876] LustreError: 112700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220326 with bad export cookie 6907000083165994499 [ 3134.986448] LustreError: 112700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3134.988246] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3141.523604] Lustre: server umount lustre-MDT0001 complete [ 3152.588410] Lustre: server umount lustre-OST0000 complete [ 3163.503677] Lustre: server umount lustre-OST0001 complete [ 3170.613339] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3180.967997] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3217.948482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.326269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3218.394319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3218.536834] Lustre: lustre-MDT0000: new disk, initializing [ 3218.653230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.659070] Lustre: Skipped 3 previous similar messages [ 3218.680748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3223.207643] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3230.743449] Lustre: server umount lustre-MDT0000 complete [ 3238.904577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.179968] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3239.194269] Lustre: 116914:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3239.206993] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3242.878611] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3247.816658] Lustre: server umount lustre-MDT0000 complete [ 3255.875487] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:27:25 (1781220445) [ 3257.149149] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3258.541626] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:27:28 (1781220448) [ 3276.928678] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3289.170585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.842758] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3301.685757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.862623] Lustre: 118933:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3301.941488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3301.945111] Lustre: Skipped 1 previous similar message [ 3302.041804] Lustre: lustre-MDT0001: new disk, initializing [ 3302.128358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3302.132017] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3306.504644] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3311.368237] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3318.218352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3318.600427] Lustre: lustre-OST0000: new disk, initializing [ 3318.606591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3318.625027] Lustre: 120476:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3319.878920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3319.897995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3320.082078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3324.831265] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3335.671072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3335.905313] Lustre: lustre-OST0001: new disk, initializing [ 3335.911507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3335.921370] Lustre: 121331:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3337.556772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3337.567533] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3337.689868] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3342.334566] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3352.227473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3355.959858] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3737.584781] Lustre: 127726:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3737.596161] Lustre: 127726:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3737.603623] Lustre: 127726:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3737.609375] Lustre: 127726:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3737.618382] Lustre: 127726:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3737.624266] Lustre: 127726:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4048.207798] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:40:37 (1781221237) [ 4064.739290] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.742289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4064.754168] Lustre: Skipped 8 previous similar messages [ 4064.765198] Lustre: Skipped 9 previous similar messages [ 4069.635551] Lustre: server umount lustre-MDT0000 complete [ 4069.860419] LustreError: 122124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.885185] LustreError: 122124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4073.448617] LustreError: 127752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221264 with bad export cookie 6907000083166063078 [ 4073.454132] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4073.464092] LustreError: 127752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4074.976409] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.987511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4074.992554] Lustre: Skipped 1 previous similar message [ 4075.015836] Lustre: Skipped 1 previous similar message [ 4080.039194] Lustre: server umount lustre-MDT0001 complete [ 4089.515361] Lustre: server umount lustre-OST0000 complete [ 4099.292329] Lustre: server umount lustre-OST0001 complete [ 4105.554608] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4116.072336] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4151.564566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.887857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4151.923406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4152.013302] Lustre: lustre-MDT0000: new disk, initializing [ 4152.123494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.132857] Lustre: Skipped 5 previous similar messages [ 4152.153549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4156.532502] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4167.439572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.632325] Lustre: 130753:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4167.719220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4167.723901] Lustre: Skipped 1 previous similar message [ 4167.825332] Lustre: lustre-MDT0001: new disk, initializing [ 4167.980748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4168.002123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4171.985080] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4181.119520] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4186.011813] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4194.421875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4194.680622] Lustre: lustre-OST0000: new disk, initializing [ 4194.689036] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4194.707314] Lustre: 132496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4194.884567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.901767] Lustre: Skipped 1 previous similar message [ 4196.546963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4196.565773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4196.698437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4201.341105] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4208.501565] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4216.687595] Lustre: lustre-MDD0000: changelog on [ 4219.581390] Lustre: lustre-MDD0001: changelog on [ 6262.282889] Lustre: lustre-MDD0001: changelog off [ 6264.358467] Lustre: 130761:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6265.374837] Lustre: lustre-MDD0000: changelog off [ 6275.554300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.565562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6275.585340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.592321] Lustre: Skipped 3 previous similar messages [ 6280.675419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6280.684436] Lustre: Skipped 1 previous similar message [ 6285.279298] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6285.462962] Lustre: server umount lustre-OST0000 complete [ 6290.913964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.919375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6290.926367] Lustre: Skipped 2 previous similar messages [ 6290.946458] Lustre: Skipped 1 previous similar message [ 6296.036330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6296.047878] Lustre: Skipped 1 previous similar message [ 6303.199270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6303.626902] Lustre: server umount lustre-MDT0000 complete [ 6305.249094] LustreError: 131882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.277340] LustreError: 131882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6308.145683] LustreError: 130746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223499 with bad export cookie 6907000083166879593 [ 6308.156915] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.169070] LustreError: 130746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6308.863754] Lustre: server umount lustre-MDT0001 complete [ 6322.087263] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6325.820921] Key type lgssc unregistered [ 6326.285244] LNet: 141530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6326.301512] LNetError: 141530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6327.335985] LNet: Removed LNI 192.168.204.123@tcp [ 6328.307185] Key type .llcrypt unregistered [ 6328.309187] Key type ._llcrypt unregistered [ 6350.851678] Key type ._llcrypt registered [ 6350.854697] Key type .llcrypt registered [ 6350.997869] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 6370.743445] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6372.236830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6372.247964] alg: No test for adler32 (adler32-zlib) [ 6373.412776] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 6373.668939] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6375.367177] Key type lgssc registered [ 6376.792242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6416.963828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6417.028356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.629557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6418.682741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6418.800386] Lustre: lustre-MDT0000: new disk, initializing [ 6418.915462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.949326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6423.880770] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6435.383742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.597848] Lustre: 144698:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6435.729341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6435.750354] Lustre: Skipped 1 previous similar message [ 6435.903298] Lustre: lustre-MDT0001: new disk, initializing [ 6436.099267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6436.140445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6436.157812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6440.277790] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6448.219138] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6452.185980] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6459.655089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6460.071187] Lustre: lustre-OST0000: new disk, initializing [ 6460.079928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6460.091290] Lustre: 146440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6460.246112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6466.137330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6466.155697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6466.310985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6467.383254] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6475.868419] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6482.277439] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_module obdecho/obdecho [ 6484.310902] LustreError: 147314:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6484.337246] LustreError: 147314:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6486.049030] LustreError: 147410:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6502.372212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.374799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6502.397760] Lustre: Skipped 2 previous similar messages [ 6502.412124] Lustre: Skipped 2 previous similar messages [ 6504.443878] Lustre: server umount lustre-MDT0000 complete [ 6508.679994] LustreError: 144689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223699 with bad export cookie 11877555449655037175 [ 6508.684149] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6508.689743] LustreError: 144689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6509.185634] Lustre: server umount lustre-MDT0001 complete [ 6523.923792] Lustre: server umount lustre-OST0000 complete [ 6546.860993] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 6559.709797] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6599.806434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6600.101450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6600.141746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6600.257644] Lustre: lustre-MDT0000: new disk, initializing [ 6600.379611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6600.394103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6604.806188] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6614.921203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.007237] Lustre: 151094:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6615.069505] LustreError: 151104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6615.085076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6615.092890] LustreError: 151104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6615.105113] Lustre: Skipped 1 previous similar message [ 6615.165688] Lustre: lustre-MDT0001: new disk, initializing [ 6615.285337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.334271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6615.343519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6619.748224] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6628.641421] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6632.909397] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6640.066954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6640.427508] Lustre: lustre-OST0000: new disk, initializing [ 6640.433419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6640.445772] Lustre: 152832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6640.571643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.005252] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6648.915065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6648.926379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6649.075946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6655.248809] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6660.374680] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6660.619472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6665.804747] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6666.050530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6669.813779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.818647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.840271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.853908] Lustre: Skipped 1 previous similar message [ 6674.417813] Lustre: server umount lustre-OST0000 complete [ 6680.033657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.056253] Lustre: Skipped 1 previous similar message [ 6680.063935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6685.153282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6685.168180] Lustre: Skipped 2 previous similar messages [ 6689.255721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6689.264813] Lustre: Skipped 1 previous similar message [ 6693.344236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6693.671376] Lustre: server umount lustre-MDT0000 complete [ 6694.375081] LustreError: 151104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.407704] LustreError: 151104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6698.351678] LustreError: 152831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223889 with bad export cookie 11877555449655038134 [ 6698.355471] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6698.369146] LustreError: 152831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6698.851695] Lustre: server umount lustre-MDT0001 complete [ 6703.627665] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:24:52 (1781223892) [ 6705.886465] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6708.470905] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:24:57 (1781223897) [ 6717.108959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.635551] LustreError: 155099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.691276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.964186] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6723.044103] LustreError: 155100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.151145] LustreError: 155099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6729.843464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6733.801560] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6739.000798] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6743.122895] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6750.532320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6750.917275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6750.922743] Lustre: Skipped 1 previous similar message [ 6758.279082] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6765.100271] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6770.562324] LustreError: 157338:0:(ldlm_resource.c:1180:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0xdce4c939 (0000000040a60c0b) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6771.042604] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 6771.172919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6771.190945] LustreError: Skipped 1 previous similar message [ 6771.196212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.206409] Lustre: Skipped 1 previous similar message [ 6772.774942] Lustre: server umount lustre-MDT0000 complete [ 6776.139384] LustreError: 157248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6776.806680] LustreError: 155080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223967 with bad export cookie 11877555449655039016 [ 6776.824130] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6777.833939] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.862155] Lustre: Skipped 2 previous similar messages [ 6783.248954] Lustre: server umount lustre-MDT0001 complete [ 6791.138223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6806.948530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.155310] LustreError: 158601:0:(llog.c:1646:llog_backup()) MGC192.168.204.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6807.161365] Lustre: 158601:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6807.524513] LustreError: 158606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.539905] LustreError: 158606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6843.360708] LustreError: 158606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6843.391358] LustreError: 158606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 6867.935789] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6867.946606] Lustre: 158601:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6907.361536] LustreError: 156703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.394320] LustreError: 156703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 6908.907066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6912.532243] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6914.030137] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6921.597184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.210468] LustreError: 142919:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000270fd7b8 x1867748279971840/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6927.117654] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6927.864636] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6936.096897] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6938.606667] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6943.584812] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6945.644693] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6968.801124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6968.809302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6968.817873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6968.837165] Lustre: Skipped 1 previous similar message [ 6968.850268] Lustre: Skipped 9 previous similar messages [ 6969.198575] Lustre: server umount lustre-MDT0000 complete [ 6973.154984] LustreError: 155082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224164 with bad export cookie 11877555449655045470 [ 6973.160285] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6973.168337] LustreError: 155082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6973.443185] Lustre: server umount lustre-MDT0001 complete [ 6988.844603] Lustre: server umount lustre-OST0000 complete [ 7013.315982] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7023.444711] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7063.257399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.470333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7063.495832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7063.566292] Lustre: lustre-MDT0000: new disk, initializing [ 7063.685332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7063.700430] Lustre: Skipped 1 previous similar message [ 7063.725990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7068.169830] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7079.744397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.871576] Lustre: 163453:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7079.983557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7080.259459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7086.133857] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7094.612496] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7099.008402] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7106.592979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7106.969458] Lustre: lustre-OST0000: new disk, initializing [ 7106.984193] Lustre: Skipped 1 previous similar message [ 7106.995039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7106.997753] Lustre: Skipped 2 previous similar messages [ 7107.007865] Lustre: 165196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7113.317860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7113.329200] Lustre: Skipped 1 previous similar message [ 7113.337742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7113.446073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7113.786330] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7122.564677] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7128.069878] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7128.329338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7133.685381] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7133.988885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7139.298642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7139.306256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.343133] Lustre: Skipped 2 previous similar messages [ 7142.730345] Lustre: server umount lustre-OST0000 complete [ 7149.540661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7149.548179] Lustre: Skipped 3 previous similar messages [ 7153.795609] Lustre: server umount lustre-MDT0000 complete [ 7154.663463] LustreError: 163466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.683557] LustreError: 163466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 7157.956242] LustreError: 164232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224348 with bad export cookie 11877555449655062298 [ 7157.959300] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7157.975709] LustreError: 164232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7158.470637] Lustre: server umount lustre-MDT0001 complete [ 7167.818533] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:32:37 (1781224357) [ 7175.988794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.356433] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7192.468901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.380718] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7204.957227] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7208.456313] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7214.733374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.462230] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7229.797970] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7242.724220] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:33:52 (1781224432) [ 7251.936311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7251.943146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7251.946765] LustreError: Skipped 1 previous similar message [ 7251.975490] Lustre: Skipped 3 previous similar messages [ 7254.210776] Lustre: server umount lustre-OST0000 complete [ 7265.035486] Lustre: server umount lustre-MDT0000 complete [ 7268.974927] LustreError: 168181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224460 with bad export cookie 11877555449655063173 [ 7268.975689] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7268.989809] LustreError: 168181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7269.208953] Lustre: server umount lustre-MDT0001 complete [ 7283.978923] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7288.399391] Key type lgssc unregistered [ 7288.953848] LNet: 170835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7288.959905] LNetError: 170835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7290.022962] LNet: Removed LNI 192.168.204.123@tcp [ 7291.565820] Key type .llcrypt unregistered [ 7291.567709] Key type ._llcrypt unregistered [ 7313.147954] Key type ._llcrypt registered [ 7313.152778] Key type .llcrypt registered [ 7313.312552] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7314.472890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7314.556885] alg: No test for adler32 (adler32-zlib) [ 7315.702786] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 7316.033321] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7317.799444] Key type lgssc registered [ 7319.101600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7327.339787] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7327.370119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7342.943926] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7348.066335] LustreError: 172064:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 7353.183676] LustreError: 172064:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7353.204193] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7353.210124] LustreError: MGC192.168.204.123@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7353.225758] LustreError: 172064:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7353.250602] LustreError: 172064:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7353.269093] LustreError: 172064:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7353.379861] Lustre: server umount lustre-OST0000 complete [ 7353.381931] LustreError: 172064:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7362.530158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.228598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7368.682583] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7374.829188] LustreError: 172564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.847638] LustreError: 172564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7378.117627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.497477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7383.134754] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7388.729251] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7392.131323] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7399.721157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7400.038444] LustreError: 174152:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7400.055204] Lustre: 174152:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.123@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7400.291519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7407.482350] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7414.667548] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7419.360602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7419.374683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.399416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7419.874957] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.906397] Lustre: Skipped 1 previous similar message [ 7419.928274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7419.941249] Lustre: Skipped 1 previous similar message [ 7423.720276] Lustre: server umount lustre-MDT0000 complete [ 7425.003539] LustreError: 172565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7425.017634] LustreError: 172565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7427.995471] LustreError: 172544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224619 with bad export cookie 14395038142362925969 [ 7428.019345] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.021667] LustreError: 172544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7428.779494] Lustre: server umount lustre-MDT0001 complete [ 7443.215811] Lustre: server umount lustre-OST0000 complete [ 7455.591960] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7468.332579] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7510.555550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.837434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7510.891166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7511.038548] Lustre: lustre-MDT0000: new disk, initializing [ 7511.198518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7511.245028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7514.464443] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7526.711270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.819774] Lustre: 177530:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7526.871581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7526.875393] Lustre: Skipped 1 previous similar message [ 7526.964861] Lustre: lustre-MDT0001: new disk, initializing [ 7527.090901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7527.141810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7527.155407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7532.026736] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7539.896824] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7544.429639] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7551.872332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7552.288433] Lustre: lustre-OST0000: new disk, initializing [ 7552.299412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7552.311534] Lustre: 179274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7552.442547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7560.287400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7560.308336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7560.461132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7560.656069] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7569.231475] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7574.886272] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7575.151921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7580.471326] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7580.747535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7586.287243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7586.296564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.303744] LustreError: Skipped 1 previous similar message [ 7586.307216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7586.328692] Lustre: Skipped 1 previous similar message [ 7586.374302] Lustre: Skipped 1 previous similar message [ 7589.159655] Lustre: server umount lustre-OST0000 complete [ 7593.952814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7593.962876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.982110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.636075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.659850] Lustre: Skipped 3 previous similar messages [ 7607.263302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7607.511569] Lustre: server umount lustre-MDT0000 complete [ 7607.782355] LustreError: 177536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7607.804314] LustreError: 177536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7611.969787] LustreError: 177521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224803 with bad export cookie 14395038142362926690 [ 7611.971802] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7611.988209] LustreError: 177521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7612.520078] Lustre: server umount lustre-MDT0001 complete [ 7622.533329] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:40:12 (1781224812) [ 7642.991925] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7646.552056] Key type lgssc unregistered [ 7646.881831] LNet: 182061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7646.893824] LNetError: 182061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7646.910168] LNet: Removed LNI 192.168.204.123@tcp [ 7647.910554] Key type .llcrypt unregistered [ 7647.913931] Key type ._llcrypt unregistered [ 7667.566931] Key type ._llcrypt registered [ 7667.571379] Key type .llcrypt registered [ 7667.724708] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7668.972318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7669.005943] alg: No test for adler32 (adler32-zlib) [ 7670.075708] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 7670.293956] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7671.975162] Key type lgssc registered [ 7673.224880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7680.983204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7681.011536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.445605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7686.864650] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7692.774147] LustreError: 183263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.812687] LustreError: 183263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7696.543961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.250830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7704.543116] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7712.386738] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7716.428926] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7724.046992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7724.315564] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7724.321677] LustreError: 184901:0:(llog.c:1687:llog_backup()) MGC192.168.204.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7724.340914] Lustre: 184901:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.123@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7724.559060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7731.521961] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7736.991649] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7742.946625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.955049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.974636] Lustre: Skipped 2 previous similar messages [ 7742.988226] Lustre: Skipped 2 previous similar messages [ 7744.033461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7744.048683] Lustre: Skipped 1 previous similar message [ 7746.059231] Lustre: server umount lustre-MDT0000 complete [ 7749.105851] LustreError: 183264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7749.126688] LustreError: 183264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7749.857618] LustreError: 183245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224940 with bad export cookie 15552549063300127020 [ 7749.871298] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.874361] LustreError: 183245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7750.200280] LustreError: 183976:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000150926db x1867749639875712/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7750.403450] Lustre: server umount lustre-MDT0001 complete [ 7764.907707] Lustre: server umount lustre-OST0000 complete [ 7774.127260] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7787.245823] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7830.510143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.753724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7830.773169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7830.820459] Lustre: lustre-MDT0000: new disk, initializing [ 7830.936406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.960138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7834.879785] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7845.966748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.153888] Lustre: 188281:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7846.221365] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7846.225087] Lustre: Skipped 1 previous similar message [ 7846.232913] LustreError: 189045:0:(llog.c:1687:llog_backup()) MGC192.168.204.123@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7846.244893] LustreError: 189045:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7846.250846] Lustre: 189045:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.123@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7846.269986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7846.274776] Lustre: Skipped 1 previous similar message [ 7846.358741] Lustre: lustre-MDT0001: new disk, initializing [ 7846.484393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7846.551195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7846.572424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7851.435754] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7859.769661] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7863.877649] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7870.876281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7871.161704] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7871.164443] Lustre: Skipped 1 previous similar message [ 7871.167429] LustreError: 189996:0:(llog.c:1687:llog_backup()) MGC192.168.204.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7871.173518] LustreError: 189996:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7871.177684] Lustre: 189996:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.123@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7871.185454] Lustre: 189996:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7871.205845] Lustre: lustre-OST0000: new disk, initializing [ 7871.209791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7871.216738] Lustre: 190021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7871.301160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7877.453092] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7879.736081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7879.749528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7879.815494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7884.509129] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7889.059936] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7889.337777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7894.165648] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7894.457351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7900.648672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7900.649839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7900.654119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7900.664400] LustreError: Skipped 1 previous similar message [ 7900.674372] Lustre: Skipped 1 previous similar message [ 7902.958519] Lustre: server umount lustre-OST0000 complete [ 7907.809375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7907.825334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.848287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.850958] Lustre: Skipped 1 previous similar message [ 7910.884205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.739347] Lustre: server umount lustre-MDT0000 complete [ 7916.011093] LustreError: 188286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.035151] LustreError: 188286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7917.817808] LustreError: 190019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225108 with bad export cookie 15552549063300127734 [ 7917.818113] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7917.822653] LustreError: 190019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7918.267123] Lustre: server umount lustre-MDT0001 complete [ 7928.241436] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:45:18 (1781225118) [ 7945.542583] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7954.422073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7954.919048] LustreError: 192985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7954.967221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7959.723436] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7960.034973] LustreError: 192986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.381236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.386784] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7976.227410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7984.631139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7993.120524] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8002.868440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8003.127563] Lustre: lustre-OST0001: new disk, initializing [ 8003.130916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8003.139125] Lustre: 195243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8003.251817] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8003.260167] Lustre: Skipped 2 previous similar messages [ 8009.124412] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8010.859649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8010.876343] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8010.979583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8020.367749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8027.694342] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8029.674603] Lustre: DEBUG MARKER: ADD OST3 [ 8035.040543] loop: module loaded [ 8035.107663] loop0: detected capacity change from 0 to 204800000 [ 8035.419149] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8035.435459] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8035.453880] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8042.435242] loop0: detected capacity change from 0 to 204800000 [ 8042.945927] Lustre: DEBUG MARKER: STOP OST3 [ 8050.032581] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8050.348399] Lustre: lustre-OST0003: new disk, initializing [ 8050.357997] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8050.379651] Lustre: 197266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8051.803537] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8052.068037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8052.079181] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8052.121329] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8053.857535] LustreError: 197388:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8054.411654] Lustre: Failing over lustre-OST0003 [ 8054.613910] Lustre: server umount lustre-OST0003 complete [ 8056.675548] LustreError: 194438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.696206] LustreError: 194438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8057.313282] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8057.322652] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8058.003978] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8066.817214] Lustre: DEBUG MARKER: START OST3 again [ 8072.694979] LustreError: 195525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8072.715398] LustreError: 195525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 8074.335536] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8074.680103] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8074.688049] Lustre: Skipped 1 previous similar message [ 8074.710318] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8076.379626] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8076.509294] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8076.515384] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8082.518561] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8105.676864] Lustre: Failing over lustre-OST0003 [ 8105.786318] Lustre: server umount lustre-OST0003 complete [ 8107.488776] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8107.500642] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8107.507823] LustreError: 194437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8107.523249] Lustre: Skipped 2 previous similar messages [ 8125.925306] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.931178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8125.965508] Lustre: Skipped 2 previous similar messages [ 8125.986586] Lustre: Skipped 5 previous similar messages [ 8131.113872] Lustre: server umount lustre-MDT0000 complete [ 8135.316064] LustreError: 197257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225326 with bad export cookie 15552549063300128623 [ 8135.317379] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8135.327843] LustreError: 197257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8136.170195] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8136.183141] Lustre: Skipped 1 previous similar message [ 8141.793588] Lustre: server umount lustre-MDT0001 complete [ 8146.542453] Lustre: server umount lustre-OST0000 complete [ 8150.513435] Lustre: server umount lustre-OST0001 complete [ 8158.443823] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8169.267311] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8211.441343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.693694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8211.729258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8211.841686] Lustre: lustre-MDT0000: new disk, initializing [ 8211.985799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8212.010636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8217.253885] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8228.980031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.114819] Lustre: 202625:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8229.216124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8229.222256] Lustre: Skipped 1 previous similar message [ 8229.329446] Lustre: lustre-MDT0001: new disk, initializing [ 8229.487771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8229.518222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8234.544927] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8242.644284] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8247.180585] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8254.285154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8254.597332] Lustre: 204368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8256.334947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8256.446607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8261.502778] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8270.061870] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8274.990388] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8275.234621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8280.208870] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8280.483085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8287.205022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8287.229851] Lustre: Skipped 3 previous similar messages [ 8287.235899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8287.238631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8287.252443] Lustre: Skipped 3 previous similar messages [ 8287.287798] LustreError: Skipped 1 previous similar message [ 8288.656436] Lustre: server umount lustre-OST0000 complete [ 8298.562537] Lustre: server umount lustre-MDT0000 complete [ 8302.486465] LustreError: 204367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225493 with bad export cookie 15552549063300130919 [ 8302.507722] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8302.888576] Lustre: server umount lustre-MDT0001 complete [ 8306.817534] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:51:36 (1781225496) [ 8324.870256] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8337.171123] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8377.179167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8377.531371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8377.575490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8377.580097] Lustre: Skipped 1 previous similar message [ 8377.695834] Lustre: lustre-MDT0000: new disk, initializing [ 8377.700478] Lustre: Skipped 1 previous similar message [ 8377.813851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8377.819761] Lustre: Skipped 1 previous similar message [ 8383.287995] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8395.373194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.496931] Lustre: 208802:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8395.842922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8401.095736] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8409.971798] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8413.799553] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8421.496662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8421.876555] Lustre: 210544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8423.859108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8428.870415] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8437.439738] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8442.790661] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8443.017861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8447.714524] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8447.916687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8454.645793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.648223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8454.660611] Lustre: Skipped 2 previous similar messages [ 8454.671037] LustreError: Skipped 1 previous similar message [ 8454.674816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8454.692107] Lustre: Skipped 5 previous similar messages [ 8456.608955] Lustre: server umount lustre-OST0000 complete [ 8467.012533] Lustre: server umount lustre-MDT0000 complete [ 8469.984675] LustreError: 208811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8470.007678] LustreError: 208811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 8471.133626] LustreError: 210543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225662 with bad export cookie 15552549063300131801 [ 8471.149868] LustreError: 210543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8471.161224] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8471.755804] Lustre: server umount lustre-MDT0001 complete [ 8481.828096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8482.554912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8482.562568] Lustre: Skipped 5 previous similar messages [ 8487.337464] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8496.003389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8501.632225] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8508.389297] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8513.387514] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8521.377731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8529.296254] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8535.805363] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8546.801050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8546.809711] LustreError: Skipped 1 previous similar message [ 8546.815677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8546.829035] Lustre: Skipped 4 previous similar messages [ 8548.361939] Lustre: server umount lustre-OST0000 complete [ 8558.966327] Lustre: server umount lustre-MDT0000 complete [ 8562.983238] LustreError: 213334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225754 with bad export cookie 15552549063300132683 [ 8562.992417] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8562.999408] LustreError: 213334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8563.352759] Lustre: server umount lustre-MDT0001 complete [ 8674.744676] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8678.024253] Key type lgssc unregistered [ 8678.278289] LNet: 215922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8678.293718] LNetError: 215922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8678.316810] LNet: Removed LNI 192.168.204.123@tcp [ 8679.448520] Key type .llcrypt unregistered [ 8679.452075] Key type ._llcrypt unregistered [ 8706.378868] Key type ._llcrypt registered [ 8706.381519] Key type .llcrypt registered [ 8706.506911] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8724.900742] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8726.102245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8726.192745] alg: No test for adler32 (adler32-zlib) [ 8727.570281] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 8727.880626] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8729.583186] Key type lgssc registered [ 8730.903188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8770.046750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8770.085251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.527325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8771.567498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8771.710684] Lustre: lustre-MDT0000: new disk, initializing [ 8771.835379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8771.860343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8776.253757] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8787.075801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8787.221580] Lustre: 219248:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8787.283717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8787.288738] Lustre: Skipped 1 previous similar message [ 8787.359826] Lustre: lustre-MDT0001: new disk, initializing [ 8787.488058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8787.549119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8787.582048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8791.920961] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8802.444786] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8807.111510] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8815.976225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8816.405307] Lustre: lustre-OST0000: new disk, initializing [ 8816.411865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8816.418254] Lustre: 220988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8816.531326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8823.406164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8823.428544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8823.540703] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8823.578373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8831.868656] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8836.994144] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8837.321230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8842.340959] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8842.641965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8849.378059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8849.380691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.389118] LustreError: Skipped 1 previous similar message [ 8849.391726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8849.404580] Lustre: Skipped 1 previous similar message [ 8849.427324] Lustre: Skipped 1 previous similar message [ 8851.379518] Lustre: server umount lustre-OST0000 complete [ 8859.105724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8859.123265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8859.142089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8862.032489] Lustre: server umount lustre-MDT0000 complete [ 8866.211112] LustreError: 219240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226057 with bad export cookie 5836251462570621217 [ 8866.228232] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8866.233106] LustreError: 219240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8866.802378] Lustre: server umount lustre-MDT0001 complete [ 8889.272368] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8902.686350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.514521] LustreError: 223772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8903.540234] LustreError: 223772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8903.602258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8908.684335] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8908.779258] LustreError: 223773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8913.888602] LustreError: 223772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8916.876536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.566234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8922.325499] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8925.741377] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8933.448881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8933.963662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.561121] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8950.213265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8950.390125] Lustre: lustre-OST0001: new disk, initializing [ 8950.394675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8950.400387] Lustre: 226029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8950.464440] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8956.146924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8956.167388] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8956.353590] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8957.289613] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8970.266560] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8981.780208] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8996.706885] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:03:06 (1781226186) [ 8999.407259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8999.424850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.445591] Lustre: Skipped 1 previous similar message [ 8999.457625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8999.465619] Lustre: Skipped 1 previous similar message [ 9002.472574] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.472725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9002.498352] Lustre: Skipped 2 previous similar messages [ 9002.517110] Lustre: Skipped 3 previous similar messages [ 9005.543790] Lustre: server umount lustre-MDT0000 complete [ 9007.586928] LustreError: 224729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9007.626547] LustreError: 224729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9009.866399] LustreError: 223752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226200 with bad export cookie 5836251462570622099 [ 9009.868368] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9009.897464] LustreError: 223752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9010.354525] Lustre: server umount lustre-MDT0001 complete [ 9024.258801] Lustre: server umount lustre-OST0000 complete [ 9038.080390] Lustre: server umount lustre-OST0001 complete [ 9044.606164] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 9056.947692] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9093.588240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.791942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9093.812315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9093.852459] Lustre: lustre-MDT0000: new disk, initializing [ 9093.932409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9093.955792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9097.688870] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9106.693645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9106.789646] Lustre: 229929:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9106.840909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9106.843732] Lustre: Skipped 1 previous similar message [ 9106.885993] Lustre: lustre-MDT0001: new disk, initializing [ 9106.975451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9106.979950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9110.393379] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9117.829855] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9121.999485] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9128.806855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.197582] Lustre: lustre-OST0000: new disk, initializing [ 9129.201162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9129.207506] Lustre: 231672:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9131.045822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9131.066600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9131.176938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9134.986708] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9142.000897] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9146.473405] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9146.690779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9150.332125] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9150.533875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9156.580927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9156.603265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9156.652218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9158.643793] Lustre: server umount lustre-OST0000 complete [ 9167.333779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.344715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9167.356950] Lustre: Skipped 2 previous similar messages [ 9167.382830] Lustre: Skipped 3 previous similar messages [ 9169.671570] Lustre: server umount lustre-MDT0000 complete [ 9172.449307] LustreError: 229941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9172.460674] LustreError: 229941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9173.706064] LustreError: 231671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226364 with bad export cookie 5836251462570624234 [ 9173.707990] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.730722] LustreError: 231671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9174.309830] Lustre: server umount lustre-MDT0001 complete [ 9183.523506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.136425] LustreError: 233740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9184.243282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9184.259409] Lustre: Skipped 2 previous similar messages [ 9188.717755] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9197.302271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9201.805744] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9206.890076] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9211.761853] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9219.298114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9227.448054] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9234.343284] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9381.858707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9381.868237] LustreError: Skipped 1 previous similar message [ 9381.873808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.891161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9384.259924] Lustre: server umount lustre-OST0000 complete [ 9402.335124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9402.491996] Lustre: server umount lustre-MDT0000 complete [ 9403.362067] LustreError: 233741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9403.393521] LustreError: 233741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9406.964754] LustreError: 233720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226597 with bad export cookie 5836251462570625116 [ 9406.979109] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9406.979436] LustreError: 233720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9407.584527] Lustre: server umount lustre-MDT0001 complete [ 9503.354709] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9508.204848] Key type lgssc unregistered [ 9508.607312] LNet: 236994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9508.628048] LNetError: 236994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9508.645917] LNet: Removed LNI 192.168.204.123@tcp [ 9509.979239] Key type .llcrypt unregistered [ 9509.983511] Key type ._llcrypt unregistered [ 9542.053920] Key type ._llcrypt registered [ 9542.061789] Key type .llcrypt registered [ 9542.186535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 9561.785872] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9562.850316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9562.895765] alg: No test for adler32 (adler32-zlib) [ 9564.372577] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 9564.745774] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9566.495282] Key type lgssc registered [ 9568.307893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9629.303887] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9640.939236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9641.013937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9642.278367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9642.348417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9642.441941] Lustre: lustre-MDT0000: new disk, initializing [ 9642.594996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9642.631281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9648.095327] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9660.383550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.560980] Lustre: 241015:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9660.619286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9660.622551] Lustre: Skipped 1 previous similar message [ 9660.699261] Lustre: lustre-MDT0001: new disk, initializing [ 9660.828212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9660.891323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9660.915835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9666.685416] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9672.107541] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9680.030526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.399945] Lustre: lustre-OST0000: new disk, initializing [ 9680.416274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9680.427731] Lustre: 242616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9680.640680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.162805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9682.175635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9682.311077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9687.298106] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9698.405461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.584663] Lustre: lustre-OST0001: new disk, initializing [ 9698.592702] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9698.601394] Lustre: 243471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9698.730553] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9705.597804] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9706.696079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9706.715747] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9706.897254] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9718.122235] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9728.661229] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9740.812717] Lustre: server umount lustre-MDT0000 complete [ 9742.816101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9742.823769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.858285] Lustre: Skipped 2 previous similar messages [ 9745.441306] LustreError: 241006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226936 with bad export cookie 16151609141335055041 [ 9745.447195] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9745.455994] LustreError: 241006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9745.834298] Lustre: server umount lustre-MDT0001 complete [ 9760.608156] Lustre: server umount lustre-OST0000 complete [ 9763.424828] Lustre: 238536:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226938/real 1781226938] req@ffff94be08338f00 x1867751625930752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9763.484025] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.492754] Lustre: Skipped 1 previous similar message [ 9766.047141] Lustre: 238537:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226941/real 1781226941] req@ffff94bf3b005680 x1867751625931008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9767.045863] Lustre: server umount lustre-OST0001 complete [ 9784.027160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9793.937598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9803.917213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9813.338916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9838.320443] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9850.200275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9850.236929] Lustre: lustre-MDT0000: reset Object Index mappings [ 9851.036958] LustreError: 248447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.068326] LustreError: 248447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9851.161260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9856.492583] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9857.845530] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9861.609156] LustreError: 248447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9866.737417] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.066220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9869.080453] Lustre: lustre-MDT0001: reset Object Index mappings [ 9869.457774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9874.474281] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9877.896694] Lustre: 249556:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9887.004977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9887.539257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9894.702280] LustreError: 249910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.834281] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9896.792684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9904.504375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9910.254709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9911.311270] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9920.032230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9924.890572] Lustre: 251394:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9935.844349] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9935.847743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9935.848877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9935.863763] Lustre: Skipped 2 previous similar messages [ 9937.650221] Lustre: server umount lustre-MDT0000 complete [ 9940.963631] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9940.976186] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9941.977952] LustreError: 248427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227133 with bad export cookie 16151609141335061831 [ 9941.982279] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9941.999509] LustreError: 248427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9942.629766] Lustre: server umount lustre-MDT0001 complete [ 9958.416587] Lustre: server umount lustre-OST0000 complete [ 9962.463175] Lustre: 238537:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227137/real 1781227137] req@ffff94bf1977ad00 x1867751625985280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9962.501294] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9963.027946] Lustre: server umount lustre-OST0001 complete [ 9982.683123] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:19:32 (1781227172) [ 9999.244395] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [10010.672356] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10052.370280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.756195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10052.827738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10052.962947] Lustre: lustre-MDT0000: new disk, initializing [10053.095201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10053.100399] Lustre: Skipped 1 previous similar message [10053.151374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10057.874663] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10069.067484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10069.208487] Lustre: 255614:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10069.315941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10069.324213] Lustre: Skipped 1 previous similar message [10069.418523] Lustre: lustre-MDT0001: new disk, initializing [10069.614442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10069.637873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10074.391811] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10082.420251] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10086.449924] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10093.028916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10093.408976] Lustre: lustre-OST0000: new disk, initializing [10093.412889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10093.419723] Lustre: 257357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10094.961846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10094.969695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10095.121309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10099.544373] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10107.326766] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10111.807723] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10112.041628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10116.665774] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10116.966673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10120.692272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10120.695134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10120.710843] LustreError: Skipped 1 previous similar message [10120.743253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10120.748130] Lustre: Skipped 4 previous similar messages [10125.118801] Lustre: server umount lustre-OST0000 complete [10130.917604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10130.920682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10130.921865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10130.947712] Lustre: Skipped 2 previous similar messages [10135.634369] Lustre: server umount lustre-MDT0000 complete [10136.032721] LustreError: 255628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10136.050118] LustreError: 255628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10139.595987] LustreError: 256394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227330 with bad export cookie 16151609141335063924 [10139.602654] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10139.609062] LustreError: 256394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10140.065443] Lustre: server umount lustre-MDT0001 complete [10158.846614] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10168.758512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10169.147065] LustreError: 260130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10169.197696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10169.206767] Lustre: Skipped 2 previous similar messages [10174.358379] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10184.080408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10188.828179] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10191.412386] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10196.838214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10202.076547] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10210.354039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10210.512790] Lustre: lustre-OST0001: new disk, initializing [10210.518479] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10210.524646] Lustre: 262391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10211.771339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10211.788576] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10211.926602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10216.247698] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10224.602078] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10227.876248] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10232.454459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10232.616486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10240.482139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10240.487584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10240.499669] Lustre: Skipped 3 previous similar messages [10240.504017] Lustre: Skipped 3 previous similar messages [10245.606847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10245.610677] Lustre: Skipped 3 previous similar messages [10252.255839] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10252.639596] Lustre: server umount lustre-MDT0000 complete [10255.851242] LustreError: 260125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.869564] LustreError: 260125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10256.388564] LustreError: 263137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227447 with bad export cookie 16151609141335064806 [10256.414473] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10256.841175] Lustre: server umount lustre-MDT0001 complete [10270.286948] Lustre: server umount lustre-OST0000 complete [10283.357640] Lustre: server umount lustre-OST0001 complete [10297.928985] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10307.311926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.768053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10307.776672] Lustre: Skipped 3 previous similar messages [10311.998084] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10320.169137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.424098] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10327.176762] Lustre: 266517:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10332.864853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10338.749653] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10346.908305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10352.329310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10352.375299] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10353.568233] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10360.652970] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10364.550576] Lustre: 268360:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10370.005757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10370.228660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10380.164116] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:26:09 (1781227569) [10397.157340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10397.165203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10397.179489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10397.185850] Lustre: Skipped 4 previous similar messages [10401.888953] Lustre: server umount lustre-MDT0000 complete [10403.818682] LustreError: 265410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10403.838463] LustreError: 265410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10405.370285] LustreError: 268362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227596 with bad export cookie 16151609141335071239 [10405.379350] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10405.383529] LustreError: 268362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10411.909657] Lustre: server umount lustre-MDT0001 complete [10421.390138] Lustre: server umount lustre-OST0000 complete [10425.403504] Lustre: server umount lustre-OST0001 complete [10430.946398] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [10440.569974] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10483.530945] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10492.957238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10493.210756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10493.247832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10493.340557] Lustre: lustre-MDT0000: new disk, initializing [10493.436396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10497.361793] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10507.516413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10507.653740] Lustre: 272547:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10508.020134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10511.805483] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10521.928899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10522.243220] Lustre: 274146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10523.720968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10528.579569] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10538.070714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10538.256075] Lustre: 275001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10540.060906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10544.616244] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10554.658507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10558.753447] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10558.758174] Lustre: Skipped 1 previous similar message [10572.418093] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:29:22 (1781227762) [10574.315480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10574.321579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10574.330032] Lustre: Skipped 5 previous similar messages [10574.334987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10574.339888] Lustre: Skipped 8 previous similar messages [10580.573734] Lustre: server umount lustre-MDT0000 complete [10583.572919] LustreError: 275000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227774 with bad export cookie 16151609141335078995 [10583.582740] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10583.583350] LustreError: 275000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10583.923505] Lustre: server umount lustre-MDT0001 complete [10597.827989] Lustre: server umount lustre-OST0000 complete [10611.568292] Lustre: server umount lustre-OST0001 complete [10618.729408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10619.226631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10619.232336] Lustre: Skipped 7 previous similar messages [10622.794513] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10630.399759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10634.625624] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10640.285121] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10644.203495] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10650.799226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10656.745410] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10660.325552] LustreError: 278834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10660.355328] LustreError: 278834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [10661.984463] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10683.816839] loop1: detected capacity change from 0 to 204800000 [10684.138949] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10684.150226] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10684.171403] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10690.554691] loop1: detected capacity change from 0 to 204800000 [10696.554651] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.686892] LustreError: 277220:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs24918: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10696.703976] LustreError: 277220:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs24918 [10696.711930] LustreError: 277220:0:(mgs_handler.c:517:mgs_target_reg()) fs24918-MDT0001: Error checking target index: rc = -13 [10696.723807] LustreError: fs24918-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10696.738873] LustreError: 280610:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10696.745985] LustreError: 280610:0:(tgt_mount.c:2084:server_put_super()) no obd fs24918-MDT0001 [10696.751915] LustreError: 280610:0:(tgt_mount.c:116:server_deregister_mount()) fs24918-MDT0001 not registered [10696.802324] Lustre: server umount fs24918-MDT0001 complete [10696.808279] LustreError: 280610:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10706.122932] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10706.237838] Lustre: Setting parameter fs24918-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs24918-MDT0001 [10706.276627] Lustre: srv-fs24918-MDT0001: No data found on store. Initialize space. [10706.279697] Lustre: Skipped 4 previous similar messages [10706.344568] Lustre: fs24918-MDT0001: new disk, initializing [10706.347686] Lustre: Skipped 3 previous similar messages [10706.469156] Lustre: srv-fs24918-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10706.479348] Lustre: 281077:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24918-MDT0001-mdtlov: can't create new llog: rc = -115 [10706.489625] LustreError: 281077:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24918-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10710.929140] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10717.957680] Lustre: server umount fs24918-MDT0001 complete [10729.739309] loop1: detected capacity change from 0 to 204800000 [10734.576140] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.902605] Lustre: 282627:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24918-MDT0001-mdtlov: can't create new llog: rc = -115 [10734.911928] LustreError: 282627:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24918-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10738.684188] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10741.235958] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10743.053347] Lustre: server umount fs24918-MDT0001 complete [10754.016242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10754.027098] Lustre: Skipped 3 previous similar messages [10754.033140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10754.038659] Lustre: Skipped 4 previous similar messages [10758.815425] Lustre: server umount lustre-MDT0000 complete [10767.093473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10767.218049] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10771.144745] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10772.977801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10777.657962] loop1: detected capacity change from 0 to 204800000 [10782.859955] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.533064] Lustre: 285039:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24918-MDT0001-mdtlov: can't create new llog: rc = -115 [10783.546713] LustreError: 285039:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24918-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10788.716128] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10794.170481] Lustre: server umount fs24918-MDT0001 complete [10818.528563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10818.539959] LustreError: Skipped 2 previous similar messages [10818.546171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10818.549330] Lustre: Skipped 4 previous similar messages [10822.461858] Lustre: server umount lustre-OST0000 complete [10831.597990] Lustre: server umount lustre-MDT0000 complete [10834.890110] LustreError: 277955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228025 with bad export cookie 16151609141335081676 [10834.892522] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10834.905091] LustreError: 277955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10835.323438] Lustre: server umount lustre-MDT0001 complete [10843.811362] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10845.996273] Key type lgssc unregistered [10846.227483] LNet: 287349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10846.237940] LNetError: 287349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10846.250483] LNet: Removed LNI 192.168.204.123@tcp [10846.902298] Key type .llcrypt unregistered [10846.905270] Key type ._llcrypt unregistered [10864.970808] Key type ._llcrypt registered [10864.973514] Key type .llcrypt registered [10865.091757] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10865.877901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10865.954373] alg: No test for adler32 (adler32-zlib) [10866.950632] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [10867.113691] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [10868.730394] Key type lgssc registered [10869.730505] Lustre: Echo OBD driver; http://www.lustre.org/ [10876.527936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10876.556873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10878.074226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10882.386918] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10888.681542] LustreError: 288603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.703884] LustreError: 288603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10891.203719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10891.652980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10896.118719] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10902.010219] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10905.708242] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10911.610626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10912.165148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10917.295659] LustreError: 290203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.479423] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10922.482168] LustreError: 290205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10923.671339] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10925.260507] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10925.270389] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10930.517667] LustreError: 290205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.546732] LustreError: 290205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10938.852138] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.864254] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10941.228249] loop1: detected capacity change from 0 to 204800000 [10941.456365] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10941.468660] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10941.476597] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10947.219306] loop1: detected capacity change from 0 to 204800000 [10953.007532] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10953.198367] Lustre: fs19372-OST0000: new disk, initializing [10953.210976] Lustre: srv-fs19372-OST0000: No data found on store. Initialize space. [10953.232970] Lustre: 291995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10953.327896] Lustre: fs19372-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10956.097367] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10956.124165] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10959.671808] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10967.768489] Lustre: server umount fs19372-OST0000 complete [10979.466586] loop1: detected capacity change from 0 to 204800000 [10985.302473] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10985.536575] Lustre: fs19372-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10990.067772] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.092103] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [10991.876432] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11000.299033] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11000.854291] loop2: detected capacity change from 0 to 204800000 [11001.122868] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11001.135716] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11001.156968] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11005.420522] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11005.829895] loop2: detected capacity change from 0 to 204800000 [11010.318037] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11010.408512] LustreError: fs19372-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11010.421091] LustreError: fs19372-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11010.427161] LustreError: 295003:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11010.432446] LustreError: 295003:0:(tgt_mount.c:2084:server_put_super()) no obd fs19372-OST0001 [11010.440982] LustreError: 295003:0:(tgt_mount.c:116:server_deregister_mount()) fs19372-OST0001 not registered [11010.473080] Lustre: server umount fs19372-OST0001 complete [11010.475296] LustreError: 295003:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11017.835254] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11018.001420] Lustre: fs19372-OST0001: new disk, initializing [11018.005292] Lustre: srv-fs19372-OST0001: No data found on store. Initialize space. [11018.015404] Lustre: 295555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11018.113372] Lustre: fs19372-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11022.830408] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11030.369432] Lustre: server umount fs19372-OST0000 complete [11038.230331] Lustre: server umount fs19372-OST0001 complete [11048.616487] loop1: detected capacity change from 0 to 204800000 [11053.896293] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11054.057091] Lustre: fs19372-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11056.616136] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.633826] LustreError: 290204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11059.283347] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11064.110824] loop2: detected capacity change from 0 to 204800000 [11068.662952] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11073.685461] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11081.719154] Lustre: server umount fs19372-OST0000 complete [11089.551821] Lustre: server umount fs19372-OST0001 complete [11101.665773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11101.671208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11101.682222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11102.695341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11102.700885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11102.714679] Lustre: Skipped 1 previous similar message [11105.968768] Lustre: server umount lustre-OST0000 complete [11111.909446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11111.916253] LustreError: Skipped 1 previous similar message [11111.920045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11111.933245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11115.576299] Lustre: server umount lustre-MDT0000 complete [11118.839033] LustreError: 295554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228309 with bad export cookie 1483016806692053628 [11118.845596] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11118.857058] LustreError: 295554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11119.198902] Lustre: server umount lustre-MDT0001 complete [11127.278672] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [11129.423489] Key type lgssc unregistered [11129.620844] LNet: 301408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11129.627741] LNetError: 301408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11129.637756] LNet: Removed LNI 192.168.204.123@tcp [11130.230215] Key type .llcrypt unregistered [11130.232991] Key type ._llcrypt unregistered [11157.322623] Key type ._llcrypt registered [11157.331216] Key type .llcrypt registered [11157.454264] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [11171.847392] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [11172.788075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11172.871035] alg: No test for adler32 (adler32-zlib) [11173.924943] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [11174.144761] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [11175.783239] Key type lgssc registered [11176.678689] Lustre: Echo OBD driver; http://www.lustre.org/ [11207.709642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11207.727563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11208.952358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11209.000461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11209.123496] Lustre: lustre-MDT0000: new disk, initializing [11209.244858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11209.281679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11213.016175] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11222.495758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.599797] Lustre: 305232:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11222.672297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11222.677116] Lustre: Skipped 1 previous similar message [11222.740380] Lustre: lustre-MDT0001: new disk, initializing [11222.806097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11222.841961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11222.847226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11226.673448] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11234.187925] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11237.716606] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11243.769563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11243.986298] Lustre: lustre-OST0000: new disk, initializing [11243.990640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11243.998780] Lustre: 306974:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11244.076583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11249.229596] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11251.762358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11251.773295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11251.924610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11255.560493] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11257.483115] Lustre: Mounted lustre-client [11258.215957] Lustre: Unmounted lustre-client [11267.552337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11267.555623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11267.562509] LustreError: Skipped 1 previous similar message [11267.566453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11267.585882] Lustre: Skipped 1 previous similar message [11267.608308] Lustre: Skipped 1 previous similar message [11268.068519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11268.072821] Lustre: Skipped 1 previous similar message [11273.187775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11273.191883] Lustre: Skipped 1 previous similar message [11273.818830] Lustre: server umount lustre-OST0000 complete [11278.310195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.311631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11278.322045] Lustre: Skipped 1 previous similar message [11278.327872] Lustre: Skipped 1 previous similar message [11283.430824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11283.448200] Lustre: Skipped 1 previous similar message [11292.127125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11292.334724] Lustre: server umount lustre-MDT0000 complete [11295.504429] LustreError: 305224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228486 with bad export cookie 7719778846106710025 [11295.507208] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11295.515281] LustreError: 305224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11295.797674] Lustre: server umount lustre-MDT0001 complete [11303.825494] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [11306.162517] Key type lgssc unregistered [11317.663590] LNet: 309004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11317.672164] LNetError: 309004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11388.327042] LNet: Removed LNI 192.168.204.123@tcp [11390.217089] Key type .llcrypt unregistered [11390.220111] Key type ._llcrypt unregistered [11412.364406] Key type ._llcrypt registered [11412.370192] Key type .llcrypt registered [11412.508733] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [11429.615740] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [11430.676372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11430.928964] alg: No test for adler32 (adler32-zlib) [11431.990688] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [11432.329527] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [11434.039184] Key type lgssc registered [11435.154284] Lustre: Echo OBD driver; http://www.lustre.org/ [11464.563565] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:44:14 (1781228654) [11471.060810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11471.098397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.507744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11472.587190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11472.663424] Lustre: lustre-MDT0000: new disk, initializing [11472.796954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11472.815577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11477.126988] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11487.748248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11487.865628] Lustre: 312199:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11487.953640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11487.958717] Lustre: Skipped 1 previous similar message [11488.070772] Lustre: lustre-MDT0001: new disk, initializing [11488.171720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11488.245801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11488.256675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11492.813866] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11499.792499] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11502.887361] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11508.564830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11508.801615] Lustre: lustre-OST0000: new disk, initializing [11508.804065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11508.808944] Lustre: 313940:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11508.891974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11514.040084] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11515.489039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11515.508572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11515.629550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11521.438638] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11531.233290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11531.237772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11531.245406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11531.257992] Lustre: Skipped 1 previous similar message [11531.263942] LustreError: Skipped 1 previous similar message [11536.361482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11536.368427] Lustre: Skipped 2 previous similar messages [11537.430965] Lustre: server umount lustre-OST0000 complete [11541.484605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11541.487712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11541.496488] Lustre: Skipped 1 previous similar message [11541.508203] Lustre: Skipped 1 previous similar message [11546.592758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11546.597493] Lustre: Skipped 1 previous similar message [11547.333748] Lustre: server umount lustre-MDT0000 complete [11550.863500] LustreError: 312191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228741 with bad export cookie 3306137250121291883 [11550.868073] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11550.876270] LustreError: 312191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11551.191477] Lustre: server umount lustre-MDT0001 complete [11560.728474] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [11563.509553] Key type lgssc unregistered [11563.787821] LNet: 315568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11563.802616] LNetError: 315568:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11563.834179] LNet: Removed LNI 192.168.204.123@tcp [11564.693473] Key type .llcrypt unregistered [11564.697075] Key type ._llcrypt unregistered [11589.622692] Key type ._llcrypt registered [11589.625706] Key type .llcrypt registered [11589.754692] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [11591.014423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11591.044709] alg: No test for adler32 (adler32-zlib) [11592.132719] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [11592.350536] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [11594.040546] Key type lgssc registered [11595.182289] Lustre: Echo OBD driver; http://www.lustre.org/ [11603.037986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11603.057770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.644981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11609.078644] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11614.178272] LustreError: 316975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11614.193908] LustreError: 316975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11617.710059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11618.084592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11622.503611] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11626.987851] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11630.495390] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11636.790284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.099607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11643.701221] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11649.674946] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11657.659104] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11659.400229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11664.356422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11664.371309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11664.387286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11667.804125] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [11667.811666] Lustre: Skipped 2 previous similar messages [11669.619747] Lustre: server umount lustre-MDT0000 complete [11669.990520] LustreError: 316975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.891232] LustreError: 316970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.906857] LustreError: 316970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11673.365362] LustreError: 316956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228864 with bad export cookie 5779673526606529250 [11673.370895] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11673.382131] LustreError: 316956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11675.104760] LustreError: 316969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.109415] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.145787] Lustre: Skipped 2 previous similar messages [11675.159476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11678.050879] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [11679.995555] Lustre: server umount lustre-MDT0001 complete [11684.175085] Lustre: server umount lustre-OST0000 complete [11693.547422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11698.638415] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11701.313908] Lustre: 320701:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.123@tcp [11703.642206] Lustre: server umount MGS complete [11711.945602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11712.501375] LustreError: 321272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11712.527444] LustreError: 321272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11712.586748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11717.271969] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11722.724423] LustreError: 321273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11722.740083] LustreError: 321273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11726.395111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11726.768677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11730.745863] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11739.273611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11739.672518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11746.025764] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11751.504917] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11753.224343] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11756.850546] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11760.593760] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [11767.283973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11767.286226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.289303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11767.289313] Lustre: Skipped 1 previous similar message [11767.303446] LustreError: Skipped 1 previous similar message [11767.330332] Lustre: Skipped 1 previous similar message [11771.326792] Lustre: server umount lustre-OST0000 complete [11777.506150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11777.509098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11777.540601] Lustre: Skipped 1 previous similar message [11777.553894] Lustre: Skipped 2 previous similar messages [11781.918398] Lustre: server umount lustre-MDT0000 complete [11782.624185] LustreError: 322279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11782.652740] LustreError: 322279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11786.084642] LustreError: 321252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228977 with bad export cookie 5779673526606530090 [11786.089828] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11786.113668] LustreError: 321252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11786.589891] Lustre: server umount lustre-MDT0001 complete [11799.933836] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [11803.090240] Key type lgssc unregistered [11803.451310] LNet: 324450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11803.483366] LNetError: 324450:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11803.498556] LNet: Removed LNI 192.168.204.123@tcp [11804.363102] Key type .llcrypt unregistered [11804.365623] Key type ._llcrypt unregistered [11822.269717] Key type ._llcrypt registered [11822.273565] Key type .llcrypt registered [11822.432783] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [11837.524591] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [11838.730417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11838.893954] alg: No test for adler32 (adler32-zlib) [11840.069050] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [11840.430238] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [11842.144785] Key type lgssc registered [11843.304916] Lustre: Echo OBD driver; http://www.lustre.org/ [11879.997553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11880.044468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11881.381922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11881.429957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11881.538269] Lustre: lustre-MDT0000: new disk, initializing [11881.722174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11881.743402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11885.665622] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11897.561949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11897.754469] Lustre: 327580:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11897.844638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11897.850694] Lustre: Skipped 1 previous similar message [11897.938330] Lustre: lustre-MDT0001: new disk, initializing [11898.099601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11898.163241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11898.178838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11902.761671] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11910.580412] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11914.644665] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11921.624546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11921.978496] Lustre: lustre-OST0000: new disk, initializing [11921.981395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11921.986144] Lustre: 329321:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11922.062942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11927.719832] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11929.169628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11929.181114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11929.355175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11935.677708] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11940.101856] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11940.354493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11944.999679] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11945.244155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11950.059399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11950.069215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11950.081804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11950.090532] LustreError: Skipped 1 previous similar message [11953.415090] Lustre: server umount lustre-OST0000 complete [11959.781199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11959.791857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.801631] Lustre: Skipped 1 previous similar message [11959.808342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11959.814251] Lustre: Skipped 1 previous similar message [11963.321823] Lustre: server umount lustre-MDT0000 complete [11966.898307] LustreError: 327570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229157 with bad export cookie 8995771186310826596 [11966.903733] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11966.910252] LustreError: 327570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11967.294322] Lustre: server umount lustre-MDT0001 complete [11976.792584] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:52:46 (1781229166) [11994.446669] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [12006.684459] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [12044.996709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.322357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12045.358831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12045.455791] Lustre: lustre-MDT0000: new disk, initializing [12045.575878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.611612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12050.860062] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12064.417889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12064.623413] Lustre: 333899:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12064.671886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12064.676144] Lustre: Skipped 1 previous similar message [12064.737350] Lustre: lustre-MDT0001: new disk, initializing [12064.833615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12064.882390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12064.898737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12071.220295] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12082.142625] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12086.846201] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12094.429926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.757751] Lustre: lustre-OST0000: new disk, initializing [12094.764533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12094.774758] Lustre: 335642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12094.920110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12102.499491] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12102.747630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12102.757723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12103.001556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12111.273945] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12149.216639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12149.219106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.221879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.250928] Lustre: Skipped 2 previous similar messages [12149.259283] Lustre: Skipped 2 previous similar messages [12154.037928] Lustre: server umount lustre-OST0000 complete [12159.460559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12159.470951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12159.470959] Lustre: Skipped 1 previous similar message [12159.497534] Lustre: Skipped 1 previous similar message [12164.501217] Lustre: server umount lustre-MDT0000 complete [12164.578679] LustreError: 336357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12164.602468] LustreError: 336357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12168.539617] LustreError: 334678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229359 with bad export cookie 8995771186310827478 [12168.549695] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12168.560392] LustreError: 334678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12168.931303] Lustre: server umount lustre-MDT0001 complete [12180.431459] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12184.111261] Key type lgssc unregistered [12184.625197] LNet: 337687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12184.644539] LNetError: 337687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12184.670753] LNet: Removed LNI 192.168.204.123@tcp [12185.610883] Key type .llcrypt unregistered [12185.613071] Key type ._llcrypt unregistered [12279.052293] Key type ._llcrypt registered [12279.055195] Key type .llcrypt registered [12279.174303] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12279.474466] Key type .llcrypt unregistered [12279.480226] Key type ._llcrypt unregistered [12362.684153] Key type ._llcrypt registered [12362.688318] Key type .llcrypt registered [12362.810809] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12363.041422] Key type .llcrypt unregistered [12363.043766] Key type ._llcrypt unregistered [12395.967862] Key type ._llcrypt registered [12395.971383] Key type .llcrypt registered [12396.101779] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12396.420660] Key type .llcrypt unregistered [12396.429352] Key type ._llcrypt unregistered [12429.610127] Key type ._llcrypt registered [12429.614809] Key type .llcrypt registered [12429.747913] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12430.129652] Key type .llcrypt unregistered [12430.135495] Key type ._llcrypt unregistered [12512.702496] Key type ._llcrypt registered [12512.705592] Key type .llcrypt registered [12512.801621] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12513.040908] Key type .llcrypt unregistered [12513.043139] Key type ._llcrypt unregistered [12544.599054] Key type ._llcrypt registered [12544.601710] Key type .llcrypt registered [12544.758964] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12545.119464] Key type .llcrypt unregistered [12545.126554] Key type ._llcrypt unregistered [12575.710450] Key type ._llcrypt registered [12575.714134] Key type .llcrypt registered [12575.838438] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12576.115337] Key type .llcrypt unregistered [12576.121085] Key type ._llcrypt unregistered [12604.085409] Key type ._llcrypt registered [12604.091403] Key type .llcrypt registered [12604.255696] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12604.532152] Key type .llcrypt unregistered [12604.534495] Key type ._llcrypt unregistered [12635.182886] Key type ._llcrypt registered [12635.187942] Key type .llcrypt registered [12635.377325] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12635.811359] Key type .llcrypt unregistered [12635.823187] Key type ._llcrypt unregistered [12671.317461] Key type ._llcrypt registered [12671.320209] Key type .llcrypt registered [12671.425537] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12671.768847] Key type .llcrypt unregistered [12671.772720] Key type ._llcrypt unregistered [12737.779368] Key type ._llcrypt registered [12737.781867] Key type .llcrypt registered [12737.944568] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12738.251187] Key type .llcrypt unregistered [12738.260811] Key type ._llcrypt unregistered [12768.160551] Key type ._llcrypt registered [12768.163306] Key type .llcrypt registered [12768.324706] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12768.697579] Key type .llcrypt unregistered [12768.702859] Key type ._llcrypt unregistered [12797.470254] Key type ._llcrypt registered [12797.471942] Key type .llcrypt registered [12797.639921] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12798.208815] Key type .llcrypt unregistered [12798.212751] Key type ._llcrypt unregistered [12852.320419] Key type ._llcrypt registered [12852.322947] Key type .llcrypt registered [12852.437570] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [12868.244792] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [12869.668180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12869.689304] alg: No test for adler32 (adler32-zlib) [12870.747863] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [12871.068948] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [12872.767672] Key type lgssc registered [12874.014365] Lustre: Echo OBD driver; http://www.lustre.org/ [12912.764190] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12912.855947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12914.081039] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12914.134412] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12914.249691] Lustre: tst250fs-MDT0000: new disk, initializing [12914.321536] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12914.339589] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12918.134277] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12928.313682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12928.520160] Lustre: 357760:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12928.592914] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12928.596947] Lustre: Skipped 1 previous similar message [12928.685600] Lustre: tst250fs-MDT0001: new disk, initializing [12928.804348] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12928.856709] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12928.874419] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12933.912059] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12939.114797] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12945.902064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12946.214864] Lustre: tst250fs-OST0000: new disk, initializing [12946.218697] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12946.228165] Lustre: 359354:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12946.319676] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12946.938849] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12946.964832] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12947.113942] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12952.576916] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12964.302842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12964.507372] Lustre: tst250fs-OST0001: new disk, initializing [12964.512001] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12964.519635] Lustre: 360209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12964.624957] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12971.692398] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12974.719651] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12974.738468] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12974.849217] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12977.813960] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12980.377807] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12980.386470] Lustre: Skipped 2 previous similar messages [13000.674657] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13000.693075] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13000.710480] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13000.718559] Lustre: Skipped 2 previous similar messages [13000.740180] Lustre: Skipped 3 previous similar messages [13005.965601] Lustre: server umount tst250fs-MDT0000 complete [13010.934822] LustreError: 359361:0:(ldlm_lib.c:1179:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.975048] LustreError: 359361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [13011.538685] LustreError: 357751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230202 with bad export cookie 9528534262988897166 [13011.550308] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13011.567237] LustreError: 357751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13012.049302] Lustre: server umount tst250fs-MDT0001 complete [13026.206827] Lustre: server umount tst250fs-OST0000 complete [13040.853704] Lustre: server umount tst250fs-OST0001 complete [13057.886300] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [13066.203810] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [13100.291746] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:11:29 (1781230289) [13102.098878] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13120.736056] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [13132.522863] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [13168.010460] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12898 sec ========= 22:12:37 (1781230357) [13169.760523] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:12:39 (1781230359) === [13173.603182] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:12:43 (1781230363) === [13219.817945] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [13223.341275] Key type lgssc unregistered [13223.657465] LNet: 369242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13223.661915] LNetError: 369242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13223.696212] LNet: Removed LNI 192.168.204.123@tcp [13224.762183] Key type .llcrypt unregistered [13224.768242] Key type ._llcrypt unregistered