[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 439466429 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524592K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003061] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008362] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.023250] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028595] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.033101] *** VALIDATE proc *** [ 0.034264] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043410] debug: unmapping init [mem 0xffffffffb3059000-0xffffffffb3060fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047031] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052019] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054313] rcu: Hierarchical SRCU implementation. [ 0.056574] smp: Bringing up secondary CPUs ... [ 0.057659] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061079] smp: Brought up 1 node, 4 CPUs [ 0.062921] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133369] node 0 deferred pages initialised in 68ms [ 0.135447] devtmpfs: initialized [ 0.136240] x86/mm: Memory block size: 128MB [ 0.139708] gcov: version magic: 0x41383552 [ 0.141603] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146296] pinctrl core: initialized pinctrl subsystem [ 0.147160] [ 0.147567] ************************************************************* [ 0.149008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150008] ** ** [ 0.152009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153010] ** ** [ 0.155007] ** This means that this kernel is built to expose internal ** [ 0.157009] ** IOMMU data structures, which may compromise security on ** [ 0.158008] ** your system. ** [ 0.160009] ** ** [ 0.161006] ** If you see this message and you are not debugging the ** [ 0.162009] ** kernel, report this immediately to your vendor! ** [ 0.164009] ** ** [ 0.166013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168014] ************************************************************* [ 0.171648] NET: Registered protocol family 16 [ 0.173439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182029] cpuidle: using governor menu [ 0.183425] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184347] PCI: Using configuration type 1 for base access [ 0.186112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196088] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199063] cryptd: max_cpu_qlen set to 1000 [ 0.202044] ACPI: Added _OSI(Module Device) [ 0.204010] ACPI: Added _OSI(Processor Device) [ 0.205012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210603] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216463] ACPI: Interpreter enabled [ 0.217048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218009] ACPI: Using IOAPIC for interrupt routing [ 0.219089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.230833] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240060] acpiphp: Slot [2] registered [ 0.241062] acpiphp: Slot [5] registered [ 0.242141] acpiphp: Slot [6] registered [ 0.244134] acpiphp: Slot [7] registered [ 0.245075] acpiphp: Slot [8] registered [ 0.245979] acpiphp: Slot [9] registered [ 0.246079] acpiphp: Slot [10] registered [ 0.247129] acpiphp: Slot [3] registered [ 0.249115] acpiphp: Slot [4] registered [ 0.252126] acpiphp: Slot [11] registered [ 0.254137] acpiphp: Slot [12] registered [ 0.256105] acpiphp: Slot [13] registered [ 0.258174] acpiphp: Slot [14] registered [ 0.260117] acpiphp: Slot [15] registered [ 0.262121] acpiphp: Slot [16] registered [ 0.264106] acpiphp: Slot [17] registered [ 0.266119] acpiphp: Slot [18] registered [ 0.268117] acpiphp: Slot [19] registered [ 0.269144] acpiphp: Slot [20] registered [ 0.272138] acpiphp: Slot [21] registered [ 0.274119] acpiphp: Slot [22] registered [ 0.275151] acpiphp: Slot [23] registered [ 0.277137] acpiphp: Slot [24] registered [ 0.279121] acpiphp: Slot [25] registered [ 0.281120] acpiphp: Slot [26] registered [ 0.283114] acpiphp: Slot [27] registered [ 0.285103] acpiphp: Slot [28] registered [ 0.286094] acpiphp: Slot [29] registered [ 0.287397] acpiphp: Slot [30] registered [ 0.289115] acpiphp: Slot [31] registered [ 0.290091] PCI host bridge to bus 0000:00 [ 0.292032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.302028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.306937] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309146] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.317016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320072] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.323020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.325009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327557] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329609] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.333049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336012] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.356018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361966] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.368026] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.374014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.379017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.391023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.396027] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.400012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.404019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.413013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.419960] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.430017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.441017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.461021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.477475] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.484018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.490020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.503017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.511890] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.517010] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.521015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.532117] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.540831] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.555017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.563018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.581043] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.595308] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.598600] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.600380] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.602300] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.604116] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.607170] iommu: Default domain type: Passthrough [ 0.610364] SCSI subsystem initialized [ 0.611075] ACPI: bus type USB registered [ 0.611942] usbcore: registered new interface driver usbfs [ 0.612064] usbcore: registered new interface driver hub [ 0.614086] usbcore: registered new device driver usb [ 0.616174] pps_core: LinuxPPS API ver. 1 registered [ 0.617010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.619051] PTP clock support registered [ 0.621140] EDAC MC: Ver: 3.0.0 [ 0.622355] PCI: Using ACPI for IRQ routing [ 0.624419] NetLabel: Initializing [ 0.626012] NetLabel: domain hash size = 128 [ 0.627009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.629083] NetLabel: unlabeled traffic allowed by default [ 0.631095] vgaarb: loaded [ 0.633362] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.634013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.637207] clocksource: Switched to clocksource kvm-clock [ 0.739676] VFS: Disk quotas dquot_6.6.0 [ 0.741015] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.743759] *** VALIDATE ramfs *** [ 0.745022] *** VALIDATE hugetlbfs *** [ 0.746323] pnp: PnP ACPI init [ 0.748236] pnp: PnP ACPI: found 6 devices [ 0.763658] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.766386] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.767821] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.769502] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.771236] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.772972] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.775202] NET: Registered protocol family 2 [ 0.776938] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.781721] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.784735] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.789611] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.792943] TCP: Hash tables configured (established 65536 bind 65536) [ 0.795801] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.798238] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.800899] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.803587] NET: Registered protocol family 1 [ 0.805555] RPC: Registered named UNIX socket transport module. [ 0.806807] RPC: Registered udp transport module. [ 0.807651] RPC: Registered tcp transport module. [ 0.808620] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.810169] NET: Registered protocol family 44 [ 0.811092] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.812748] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.814155] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.815615] PCI: CLS 0 bytes, default 64 [ 0.816669] Unpacking initramfs... [ 2.239550] debug: unmapping init [mem 0xffff8a35bcc54000-0xffff8a35bffbffff] [ 2.243398] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.245812] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.248682] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.705482] Initialise system trusted keyrings [ 2.707034] Key type blacklist registered [ 2.708659] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.716457] zbud: loaded [ 2.719654] *** VALIDATE nfs *** [ 2.720823] *** VALIDATE nfs4 *** [ 2.722438] pstore: using deflate compression [ 2.725718] Platform Keyring initialized [ 2.828171] NET: Registered protocol family 38 [ 2.829762] Key type asymmetric registered [ 2.831055] Asymmetric key parser 'x509' registered [ 2.832691] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.835352] io scheduler mq-deadline registered [ 2.837127] io scheduler kyber registered [ 2.839215] io scheduler bfq registered [ 2.841098] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.843956] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.846334] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.848364] ACPI: Power Button [PWRF] [ 2.853312] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.860492] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.872283] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.877790] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.891527] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.919340] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.952808] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.957889] Non-volatile memory driver v1.3 [ 2.959707] Linux agpgart interface v0.103 [ 2.989989] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.992087] vda: detected capacity change from 0 to 75100160 [ 3.009193] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.012515] vdb: detected capacity change from 0 to 1073741824 [ 3.033586] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.035552] vdc: detected capacity change from 0 to 2621440000 [ 3.047866] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.050445] vdd: detected capacity change from 0 to 2621440000 [ 3.064710] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.067008] vde: detected capacity change from 0 to 4294967296 [ 3.079820] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.082558] vdf: detected capacity change from 0 to 4294967296 [ 3.089091] libphy: Fixed MDIO Bus: probed [ 3.107249] usbcore: registered new interface driver usbserial_generic [ 3.109650] usbserial: USB Serial support registered for generic [ 3.111201] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.113773] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.114810] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.116418] mousedev: PS/2 mouse device common for all mice [ 3.118814] rtc_cmos 00:05: RTC can wake from S4 [ 3.120556] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.121514] rtc_cmos 00:05: registered as rtc0 [ 3.125329] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.127013] intel_pstate: CPU model not supported [ 3.129809] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.131687] hid: raw HID events driver (C) Jiri Kosina [ 3.133097] usbcore: registered new interface driver usbhid [ 3.134223] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.136419] usbhid: USB HID core driver [ 3.139411] drop_monitor: Initializing network drop monitor service [ 3.141714] Initializing XFRM netlink socket [ 3.142974] NET: Registered protocol family 10 [ 3.145341] Segment Routing with IPv6 [ 3.146404] NET: Registered protocol family 17 [ 3.148235] mpls_gso: MPLS GSO support [ 3.152915] RAS: Correctable Errors collector initialized. [ 3.154462] AVX version of gcm_enc/dec engaged. [ 3.155297] AES CTR mode by8 optimization enabled [ 3.208365] sched_clock: Marking stable (3208347803, 0)->(4049009481, -840661678) [ 3.210852] registered taskstats version 1 [ 3.212248] Loading compiled-in X.509 certificates [ 3.213628] zswap: loaded using pool lzo/zbud [ 3.231300] Key type big_key registered [ 3.241477] Key type encrypted registered [ 3.242560] ima: No TPM chip found, activating TPM-bypass! [ 3.243672] ima: Allocated hash algorithm: sha1 [ 3.244641] ima: No architecture policies found [ 3.245686] evm: Initialising EVM extended attributes: [ 3.246762] evm: security.selinux [ 3.247566] evm: security.ima [ 3.248209] evm: security.capability [ 3.248994] evm: HMAC attrs: 0x1 [ 3.250607] rtc_cmos 00:05: setting system clock to 2026-09-05 04:20:18 UTC (1788582018) [ 3.255239] debug: unmapping init [mem 0xffffffffb4003000-0xffffffffb41fffff] [ 3.257287] debug: unmapping init [mem 0xffffffffb2d82000-0xffffffffb3058fff] [ 3.263253] Write protecting the kernel read-only data: 28672k [ 3.266865] debug: unmapping init [mem 0xffffffffb1403000-0xffffffffb15fffff] [ 3.270429] debug: unmapping init [mem 0xffffffffb1d14000-0xffffffffb1dfffff] [ 3.301625] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.307400] systemd[1]: Detected virtualization kvm. [ 3.308754] systemd[1]: Detected architecture x86-64. [ 3.309901] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.332041] systemd[1]: No hostname configured. [ 3.333223] systemd[1]: Set hostname to . [ 3.334463] random: systemd: uninitialized urandom read (16 bytes read) [ 3.335763] systemd[1]: Initializing machine ID from random generator. [ 3.449960] random: systemd: uninitialized urandom read (16 bytes read) [ 3.452047] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.455382] random: systemd: uninitialized urandom read (16 bytes read) [ 3.456924] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.459854] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.012709] device-mapper: uevent: version 1.0.3 [ 4.013946] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK [ 4.390462] random: fast init done ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.755836] virtio_net virtio0 ens2: renamed from eth0 [ 4.783343] scsi host0: ata_piix [ 4.810579] scsi host1: ata_piix [ 4.813875] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.816352] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.779787] random: crng init done [ 9.783469] random: 7 urandom warning(s) missed due to ratelimiting [ 9.873034] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.526762] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.679338] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.680251] SELinux: Disabled at runtime. [ 15.822666] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.848111] systemd[1]: Detected virtualization kvm. [ 15.854686] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.485912] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.494351] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.516540] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.522947] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.528162] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.547170] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.564033] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [[ 17.864078] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.202836] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.056971] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.153792] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.643065] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.795196] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.818980] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.589855] NFS: Registering the id_resolver key type [ 26.592262] Key type id_resolver registered [ 26.594773] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg247-server login: [ 54.509525] hrtimer: interrupt took 11732209 ns [ 85.964770] libcfs: loading out-of-tree module taints kernel. [ 86.015855] Key type ._llcrypt registered [ 86.018392] Key type .llcrypt registered [ 86.130096] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 103.608613] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 105.788344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.800726] alg: No test for adler32 (adler32-zlib) [ 107.111611] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 108.067583] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 109.839216] Key type lgssc registered [ 111.850845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.911229] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.311856] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 186.818461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 186.852885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.264441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 188.338438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 188.422178] Lustre: lustre-MDT0000: new disk, initializing [ 188.509656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.523853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.930250] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.918338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.064707] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 209.094855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 209.099698] Lustre: Skipped 1 previous similar message [ 209.202357] Lustre: lustre-MDT0001: new disk, initializing [ 209.281183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.306251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.322515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.544783] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.003517] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.547678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.825785] Lustre: lustre-OST0000: new disk, initializing [ 227.829651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.832288] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.918934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.603206] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.527202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.544658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.633241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 248.079104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.248491] Lustre: lustre-OST0001: new disk, initializing [ 248.254477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 248.262767] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.337960] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 254.492793] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.539757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 255.549401] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 255.668449] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 267.051469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.779873] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.156869] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing check_logdir /tmp/testlogs/ [ 291.743292] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing yml_node [ 296.740502] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 299.984485] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 303.069688] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 304.985251] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 00:25:18 EDT 2026 [ 322.171906] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 323.882095] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 340.961455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.963224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.980401] Lustre: Skipped 3 previous similar messages [ 341.001952] Lustre: Skipped 3 previous similar messages [ 345.467469] Lustre: server umount lustre-MDT0000 complete [ 348.657481] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582363 with bad export cookie 10690603635497554368 [ 348.663280] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 348.665927] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 351.201134] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.206089] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.216399] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 351.253180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.259971] Lustre: Skipped 1 previous similar message [ 353.248812] LustreError: 9810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.251151] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 353.268109] LustreError: 9810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 353.282082] Lustre: Skipped 1 previous similar message [ 354.956462] Lustre: server umount lustre-MDT0001 complete [ 358.672234] Lustre: server umount lustre-OST0000 complete [ 362.172196] Lustre: server umount lustre-OST0001 complete [ 369.683856] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 380.620201] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 419.178606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.397987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 419.470482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 419.633927] Lustre: lustre-MDT0000: new disk, initializing [ 419.732674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.761044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 424.543353] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 435.979156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.107538] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 436.207971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 436.212922] Lustre: Skipped 1 previous similar message [ 436.299690] Lustre: lustre-MDT0001: new disk, initializing [ 436.411281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 436.458651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 436.467520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.341663] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 453.044300] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 459.626460] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 467.560104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 467.791658] Lustre: lustre-OST0000: new disk, initializing [ 467.797364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 467.804128] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 467.908291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.202360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 469.216980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 469.389287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 475.856810] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 486.060974] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 491.077927] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 491.265512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 495.990959] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.264892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.198817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.203642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.218484] Lustre: Skipped 2 previous similar messages [ 500.220449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.230237] Lustre: Skipped 1 previous similar message [ 504.606705] Lustre: server umount lustre-OST0000 complete [ 510.432743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.447288] Lustre: Skipped 1 previous similar message [ 510.447852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 515.431201] Lustre: server umount lustre-MDT0000 complete [ 515.553487] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.570682] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 519.082805] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582534 with bad export cookie 10690603635497555901 [ 519.096767] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.111249] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 519.589927] Lustre: server umount lustre-MDT0001 complete [ 525.988980] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:28:59 (1788582539) [ 546.841052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.394213] LustreError: 19922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.508669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.367910] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 552.930735] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.092107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.237555] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 574.046119] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 580.352844] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.528938] Lustre: Failing over lustre-MDT0000 [ 582.784446] Lustre: server umount lustre-MDT0000 complete [ 586.661909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.674929] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.695768] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 602.079285] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788582601/real 1788582601] req@ffff8a3504167480 x1875464288369664/t0(0) o400->MGC192.168.202.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788582617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 602.099496] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.633725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.242320] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.259897] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 613.365962] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.383524] Lustre: MGS: Client 7a353103-0cb2-4bb8-986c-57bd6810b145 (at 0@lo) reconnecting [ 613.397290] Lustre: MGC192.168.202.147@tcp: Connection restored to 0@lo (at 0@lo) [ 613.623771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.625846] Lustre: Skipped 1 previous similar message [ 613.679973] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 618.437724] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 618.991413] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 618.996590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 619.043864] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 628.081861] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 632.161909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.168481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.182464] Lustre: Skipped 2 previous similar messages [ 632.189233] Lustre: Skipped 2 previous similar messages [ 638.554385] Lustre: server umount lustre-MDT0000 complete [ 642.531053] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.547081] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582657 with bad export cookie 10690603635497557315 [ 642.564547] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 642.579677] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.590804] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 642.892577] Lustre: server umount lustre-MDT0001 complete [ 652.908913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.818580] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 668.902696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.752689] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 683.815368] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 690.948719] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 693.585450] Lustre: Failing over lustre-MDT0000 [ 694.241887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.264663] Lustre: Skipped 1 previous similar message [ 695.912353] Lustre: server umount lustre-MDT0000 complete [ 710.624078] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788582709/real 1788582709] req@ffff8a36378c5c00 x1875464288401024/t0(0) o400->MGC192.168.202.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788582725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 710.636084] LustreError: 23260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.648112] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.660662] LustreError: 23260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 716.679082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.955871] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x945ca75d8054e0f4 [ 719.983838] Lustre: MGC192.168.202.147@tcp: Connection restored to 0@lo (at 0@lo) [ 719.994452] Lustre: Skipped 1 previous similar message [ 720.313769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 724.794953] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 725.475363] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 725.497295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 725.573529] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 738.951897] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 745.441107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.448636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 745.462340] Lustre: Skipped 1 previous similar message [ 745.488125] Lustre: Skipped 3 previous similar messages [ 749.803223] Lustre: server umount lustre-MDT0000 complete [ 753.511441] LustreError: 23242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582768 with bad export cookie 10690603635497558260 [ 753.514350] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.518784] LustreError: 23242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 753.768994] Lustre: server umount lustre-MDT0001 complete [ 764.763600] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:32:58 (1788582778) [ 787.166217] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 798.737589] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 844.482835] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 857.710172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.111862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 858.144954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 858.250574] Lustre: lustre-MDT0000: new disk, initializing [ 858.385717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.394146] Lustre: Skipped 3 previous similar messages [ 858.414634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 864.171886] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 875.698519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.778539] Lustre: 29659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 875.988303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 880.831790] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 890.710970] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 896.501884] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 902.634945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 902.947331] Lustre: 31629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 904.895877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 904.929940] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 904.993988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 905.015589] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 909.232580] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 917.820424] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 926.396042] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 951.264851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.278541] LustreError: Skipped 1 previous similar message [ 951.297257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.317464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.736953] Lustre: server umount lustre-OST0000 complete [ 957.921483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 957.940268] LustreError: Skipped 1 previous similar message [ 963.745972] Lustre: server umount lustre-MDT0000 complete [ 966.629776] LustreError: 31743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 966.643629] LustreError: 31743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 967.543732] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582982 with bad export cookie 10690603635497558673 [ 967.553063] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 967.560892] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 967.953871] Lustre: server umount lustre-MDT0001 complete [ 981.211592] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 984.726732] Key type lgssc unregistered [ 985.243559] LNet: 33608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.255202] LNetError: 33608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.296288] LNet: Removed LNI 192.168.202.147@tcp [ 986.924349] Key type .llcrypt unregistered [ 986.926062] Key type ._llcrypt unregistered [ 1014.168176] Key type ._llcrypt registered [ 1014.169537] Key type .llcrypt registered [ 1014.240423] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 1015.009941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.060617] alg: No test for adler32 (adler32-zlib) [ 1016.371390] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 1016.612272] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 1018.255171] Key type lgssc registered [ 1019.579994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1026.262585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1026.285504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.732711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.966750] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1037.284209] LustreError: 35018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.310641] LustreError: 35018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1041.476744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.961281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1047.153457] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1054.403527] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1060.533865] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1068.279107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1068.823360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.187738] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1084.066726] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1087.299395] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1102.760630] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:38:36 (1788583116) [ 1105.106232] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1107.375317] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1114.430799] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:38:48 (1788583128) [ 1124.303124] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:38:58 (1788583138) [ 1125.308382] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1135.105383] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1135.114061] Lustre: Skipped 2 previous similar messages [ 1137.286526] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:39:10 (1788583150) [ 1139.886728] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1139.891196] Lustre: Skipped 2 previous similar messages [ 1147.990407] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1147.996725] Lustre: Skipped 1 previous similar message [ 1164.057929] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1164.063472] Lustre: Skipped 8 previous similar messages [ 1166.212713] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:39:39 (1788583179) [ 1169.436139] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1173.167247] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1183.685303] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:39:57 (1788583197) [ 1198.151901] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1198.157498] Lustre: Skipped 4 previous similar messages [ 1200.859483] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:40:14 (1788583214) [ 1220.576284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1220.583100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.591763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.597588] Lustre: Skipped 1 previous similar message [ 1220.612428] Lustre: Skipped 1 previous similar message [ 1225.700141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1225.708122] Lustre: Skipped 1 previous similar message [ 1230.829305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.837452] Lustre: Skipped 1 previous similar message [ 1232.351170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1232.498641] Lustre: server umount lustre-OST0000 complete [ 1241.058918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.078581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.101134] Lustre: Skipped 1 previous similar message [ 1246.177788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.215979] Lustre: Skipped 1 previous similar message [ 1252.319465] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1252.999132] Lustre: server umount lustre-MDT0000 complete [ 1256.419658] LustreError: 38199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.456275] LustreError: 38199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1256.655947] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583271 with bad export cookie 2016027095691151537 [ 1256.666160] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.672878] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1257.222711] Lustre: server umount lustre-MDT0001 complete [ 1270.547135] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 1274.556584] Key type lgssc unregistered [ 1274.911200] LNet: 42878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.916312] LNetError: 42878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.933602] LNet: Removed LNI 192.168.202.147@tcp [ 1275.978592] Key type .llcrypt unregistered [ 1275.980563] Key type ._llcrypt unregistered [ 1295.591724] Key type ._llcrypt registered [ 1295.593224] Key type .llcrypt registered [ 1295.692586] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 1311.722710] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 1313.093620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1313.246382] alg: No test for adler32 (adler32-zlib) [ 1314.210485] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 1314.399819] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 1316.063187] Key type lgssc registered [ 1317.381752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1357.360262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1357.377100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.757103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1358.775586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1358.863539] Lustre: lustre-MDT0000: new disk, initializing [ 1358.924170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.940605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1364.217981] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1376.346984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.468147] Lustre: 46032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1376.529229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1376.533882] Lustre: Skipped 1 previous similar message [ 1376.601246] Lustre: lustre-MDT0001: new disk, initializing [ 1376.686309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.725641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1376.746643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1380.417224] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1390.755515] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1398.595652] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1407.660683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.069446] Lustre: lustre-OST0000: new disk, initializing [ 1408.073113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1408.084464] Lustre: 47999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1408.207681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.249412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1414.267319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1414.371496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1415.917562] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1426.916421] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1432.087308] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1432.295499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1438.012475] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1438.294296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1445.346731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.349445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.355982] LustreError: Skipped 1 previous similar message [ 1445.357235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.392251] Lustre: Skipped 1 previous similar message [ 1446.791622] Lustre: server umount lustre-OST0000 complete [ 1453.538078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1453.541299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.556106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1453.558556] Lustre: Skipped 1 previous similar message [ 1455.585498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.585612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.573670] Lustre: server umount lustre-MDT0000 complete [ 1462.108880] LustreError: 46023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583477 with bad export cookie 3546607506465453703 [ 1462.116619] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1462.123900] LustreError: 46023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1462.658618] Lustre: server umount lustre-MDT0001 complete [ 1475.396404] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:44:48 (1788583488) [ 1482.349693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.815765] LustreError: 50460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.834189] LustreError: 50460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1482.932529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.647627] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1488.367907] LustreError: 50461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1493.476565] LustreError: 50460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.266550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.573185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1502.494860] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1512.543545] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1520.776878] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1528.546802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1528.998832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.407128] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1546.131383] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1562.868081] LustreError: 53014:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1564.399061] Lustre: Modifying parameter general.timeout=1 in log params [ 1566.023901] Lustre: Modifying parameter general.timeout=2 in log params [ 1569.139783] Lustre: Modifying parameter general.timeout=4 in log params [ 1569.147608] Lustre: Skipped 1 previous similar message [ 1574.100713] Lustre: Modifying parameter general.timeout=8 in log params [ 1574.103925] Lustre: Skipped 3 previous similar messages [ 1582.269576] Lustre: Modifying parameter general.timeout=15 in log params [ 1582.274291] Lustre: Skipped 6 previous similar messages [ 1598.308233] Lustre: Modifying parameter general.timeout=31 in log params [ 1598.312519] Lustre: Skipped 15 previous similar messages [ 1631.138946] Lustre: Modifying parameter general.timeout=64 in log params [ 1631.143835] Lustre: Skipped 32 previous similar messages [ 1702.773520] Lustre: Disabling parameter general.timeout= in log params [ 1702.780814] Lustre: Skipped 65 previous similar messages [ 1706.958738] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:48:40 (1788583720) [ 1710.694351] Lustre: 59967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1713.210262] Lustre: 60065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1715.550970] Lustre: 60163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1719.835687] Lustre: 60376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1719.848478] Lustre: 60376:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1755.168166] Lustre: 61845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1755.186040] Lustre: 61845:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1755.216357] LustreError: 61845:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1760.638979] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:49:34 (1788583774) [ 1779.170904] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.173402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1779.174033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1779.181773] Lustre: Skipped 2 previous similar messages [ 1779.212832] Lustre: Skipped 3 previous similar messages [ 1781.248101] Lustre: server umount lustre-MDT0000 complete [ 1784.296072] LustreError: 51641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.327022] LustreError: 51641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1785.663541] LustreError: 50440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583800 with bad export cookie 3546607506465454585 [ 1785.665430] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.675488] LustreError: 50440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1785.981332] Lustre: server umount lustre-MDT0001 complete [ 1801.328472] Lustre: server umount lustre-OST0000 complete [ 1830.833224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.239421] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1831.262460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1831.270760] Lustre: Skipped 22 previous similar messages [ 1831.513195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.707943] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1846.412537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.592711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1846.663480] Lustre: 64377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1853.005632] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1861.885491] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1867.474788] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1875.431655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.649719] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1876.025108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.027810] Lustre: Skipped 1 previous similar message [ 1883.214530] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1890.682628] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1892.888253] Lustre: 66946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1901.049733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.070144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.088169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1905.627717] Lustre: server umount lustre-OST0000 complete [ 1911.782394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.789506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.802766] Lustre: Skipped 2 previous similar messages [ 1911.828451] Lustre: Skipped 2 previous similar messages [ 1916.472460] Lustre: server umount lustre-MDT0000 complete [ 1916.902691] LustreError: 64390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.921472] LustreError: 64390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1920.550655] LustreError: 64371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583935 with bad export cookie 3546607506465457749 [ 1920.555578] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1920.560572] LustreError: 64371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1920.816372] Lustre: server umount lustre-MDT0001 complete [ 1933.061446] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 1935.818643] Key type lgssc unregistered [ 1936.061412] LNet: 68157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.073231] LNetError: 68157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.085451] LNet: Removed LNI 192.168.202.147@tcp [ 1936.997182] Key type .llcrypt unregistered [ 1936.998716] Key type ._llcrypt unregistered [ 1962.621455] Key type ._llcrypt registered [ 1962.624324] Key type .llcrypt registered [ 1962.750323] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 1964.224389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.256405] alg: No test for adler32 (adler32-zlib) [ 1965.352638] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 1965.622114] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 1967.369997] Key type lgssc registered [ 1968.561756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1976.937359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1976.991886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.529469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.231604] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1988.076540] LustreError: 69568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.119076] LustreError: 69568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1993.186630] LustreError: 69569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.224402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.190643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1999.093763] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2007.162720] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2013.336390] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2021.099418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.817765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.435331] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2036.443252] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2044.660826] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2053.606622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.609482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.634825] Lustre: Skipped 2 previous similar messages [ 2053.655188] Lustre: Skipped 2 previous similar messages [ 2058.722606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.500852] Lustre: server umount lustre-MDT0000 complete [ 2063.117136] LustreError: 69550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584078 with bad export cookie 18165485884510060378 [ 2063.130493] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.138877] LustreError: 69550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2063.414452] Lustre: server umount lustre-MDT0001 complete [ 2077.816309] Lustre: server umount lustre-OST0000 complete [ 2105.085120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.384620] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2105.441211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2105.654301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.524548] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2119.576558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.732142] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2119.760488] Lustre: 74438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2120.243143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2125.177325] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2132.360191] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2137.439513] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2144.608977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.775784] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2145.387507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.034560] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2160.474395] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2162.594441] Lustre: 77016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2164.485789] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2171.886722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2171.912120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.948699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.961816] Lustre: Skipped 2 previous similar messages [ 2172.390430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.857092] Lustre: server umount lustre-OST0000 complete [ 2182.641687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.652173] Lustre: Skipped 1 previous similar message [ 2182.678065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.683274] Lustre: Skipped 3 previous similar messages [ 2187.745086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.753097] Lustre: Skipped 1 previous similar message [ 2189.013284] Lustre: server umount lustre-MDT0000 complete [ 2192.888055] LustreError: 75606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.903103] LustreError: 75606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2193.727425] LustreError: 74432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584208 with bad export cookie 18165485884510061211 [ 2193.732080] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2193.737658] LustreError: 74432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2194.595994] Lustre: server umount lustre-MDT0001 complete [ 2212.185152] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 2214.626935] Key type lgssc unregistered [ 2214.965559] LNet: 78238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.988245] LNetError: 78238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.011349] LNet: Removed LNI 192.168.202.147@tcp [ 2216.221153] Key type .llcrypt unregistered [ 2216.223779] Key type ._llcrypt unregistered [ 2245.878663] Key type ._llcrypt registered [ 2245.880607] Key type .llcrypt registered [ 2246.073972] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2247.527350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2247.724778] alg: No test for adler32 (adler32-zlib) [ 2249.029630] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 2249.280921] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 2251.023214] Key type lgssc registered [ 2252.360940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2260.785571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2260.808560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.446893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.596149] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2272.741107] LustreError: 79651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.762454] LustreError: 79651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2277.632253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.862315] LustreError: 79652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.606975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.111886] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2292.712770] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2299.387741] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2307.296663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2307.515928] LustreError: 81478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.980382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.583582] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2324.099105] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2331.266874] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2332.727699] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2334.016533] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2336.851985] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2336.855719] Lustre: Skipped 1 previous similar message [ 2341.642604] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2341.646804] Lustre: Skipped 3 previous similar messages [ 2350.258791] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2350.267416] Lustre: Skipped 7 previous similar messages [ 2367.052391] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2367.055031] Lustre: Skipped 13 previous similar messages [ 2399.400670] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2399.425217] Lustre: Skipped 29 previous similar messages [ 2449.975704] Lustre: 87775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2453.078897] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:01:06 (1788584466) [ 2455.122688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2456.737856] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:01:10 (1788584470) [ 2461.673120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2461.683596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.685101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.727192] LustreError: Skipped 1 previous similar message [ 2466.784713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2466.797230] Lustre: Skipped 2 previous similar messages [ 2467.568943] Lustre: server umount lustre-OST0000 complete [ 2472.928699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2472.940135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.954762] Lustre: Skipped 1 previous similar message [ 2472.971709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.028415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.031545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.054729] Lustre: Skipped 1 previous similar message [ 2478.377368] Lustre: server umount lustre-MDT0000 complete [ 2482.144169] LustreError: 79652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.170694] LustreError: 79652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2482.329210] LustreError: 82240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584497 with bad export cookie 3661107254377049354 [ 2482.336225] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.346929] LustreError: 82240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2482.702555] Lustre: server umount lustre-MDT0001 complete [ 2492.940270] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 2495.702700] Key type lgssc unregistered [ 2495.964766] LNet: 89026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2495.968501] LNetError: 89026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.985703] LNet: Removed LNI 192.168.202.147@tcp [ 2496.770134] Key type .llcrypt unregistered [ 2496.772328] Key type ._llcrypt unregistered [ 2501.419171] Key type ._llcrypt registered [ 2501.421433] Key type .llcrypt registered [ 2501.586070] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2506.261042] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules [ 2507.428767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.546205] alg: No test for adler32 (adler32-zlib) [ 2508.576473] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 2508.813670] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 2510.486475] Key type lgssc registered [ 2511.934553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.487079] LustreError: 90224:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2532.799812] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2535.471197] LustreError: 90224:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2543.004777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2543.026456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.710286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.336860] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2553.830554] LustreError: 91241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.860617] LustreError: 91241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2558.957652] LustreError: 91240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.659237] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:02:55 (1788584575) [ 2564.067449] LustreError: 91241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.428882] LustreError: 91272:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2567.442924] LustreError: 91272:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2567.450156] LustreError: 91272:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2567.781945] Lustre: server umount lustre-MDT0000 complete [ 2582.400498] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 2585.671330] Key type lgssc unregistered [ 2586.134333] LNet: 92764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2586.144977] LNetError: 92764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2586.173802] LNet: Removed LNI 192.168.202.147@tcp [ 2587.164604] Key type .llcrypt unregistered [ 2587.166727] Key type ._llcrypt unregistered [ 2607.834602] Key type ._llcrypt registered [ 2607.836712] Key type .llcrypt registered [ 2607.945102] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2609.047973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2609.063683] alg: No test for adler32 (adler32-zlib) [ 2610.433599] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 2610.674277] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 2612.415203] Key type lgssc registered [ 2613.460184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2622.143861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2622.174640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.693500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.293948] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2633.189727] LustreError: 93978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.206982] LustreError: 93978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2637.706791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.965155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2645.450325] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2656.054729] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2661.099535] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2670.033154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.917454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.458903] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2686.696669] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2745.436398] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:05:58 (1788584758) [ 2747.988341] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2750.183781] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:06:03 (1788584763) [ 2760.161302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.178067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.179167] Lustre: Skipped 2 previous similar messages [ 2760.200881] Lustre: Skipped 2 previous similar messages [ 2763.451484] Lustre: server umount lustre-MDT0000 complete [ 2765.294695] LustreError: 96844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.300527] LustreError: 96844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2767.617303] LustreError: 93960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584782 with bad export cookie 15684777630858054293 [ 2767.640529] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.082976] Lustre: server umount lustre-MDT0001 complete [ 2781.167773] Lustre: server umount lustre-OST0000 complete [ 2795.318158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.118740] LustreError: 98280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.357068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.642402] LustreError: 98281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.119849] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2806.761065] LustreError: 98280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.562325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.033793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.269881] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2829.483973] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2837.229812] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2852.453646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.610969] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2852.617225] LustreError: 98269:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2852.636269] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2852.646415] LustreError: 100366:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2852.652640] LustreError: 100366:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2852.656415] LustreError: 100366:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2852.723703] Lustre: server umount lustre-OST0000 complete [ 2852.729724] LustreError: 100366:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2862.477404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.684216] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2862.695192] LustreError: 98269:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2862.728553] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2862.741730] LustreError: 100832:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2862.753085] LustreError: 100832:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2862.756870] LustreError: 100832:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2862.854733] Lustre: server umount lustre-OST0000 complete [ 2862.857038] LustreError: 100832:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2874.395576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2874.659326] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2874.663297] Lustre: Found index 0 for lustre-OST0000, updating log [ 2874.675477] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2874.745487] Lustre: lustre-OST0000: new disk, initializing [ 2874.748661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2874.753853] Lustre: 101368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2875.010183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.126630] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2886.055136] Lustre: Failing over lustre-OST0000 [ 2886.483232] Lustre: server umount lustre-OST0000 complete [ 2891.746527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.749898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.782959] Lustre: Skipped 1 previous similar message [ 2891.799741] Lustre: Skipped 1 previous similar message [ 2896.865516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.877225] Lustre: Skipped 1 previous similar message [ 2897.728181] Lustre: server umount lustre-MDT0000 complete [ 2901.983631] Lustre: 93288:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788584901/real 1788584901] req@0000000012200278 x1875466911849728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788584917 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2901.994510] LustreError: 98280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.034140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.093344] LustreError: 98280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2902.644837] LustreError: 98260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584917 with bad export cookie 15684777630858057030 [ 2902.665330] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2902.666750] LustreError: 98260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2903.079240] Lustre: server umount lustre-MDT0001 complete [ 2913.803571] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:08:47 (1788584927) [ 2933.439895] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2946.176598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.503272] LustreError: 103716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.636561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.067605] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2964.560169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.111568] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2974.991042] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2984.381331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.438657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2985.449444] Lustre: Skipped 1 previous similar message [ 2985.458702] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2990.574925] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2990.613255] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2990.614470] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2992.907088] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3003.721165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3003.867808] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3004.051981] Lustre: lustre-OST0001: new disk, initializing [ 3004.058567] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3004.075944] Lustre: 106035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3012.739597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3012.754178] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3012.892038] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3013.272837] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3022.684610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3031.215900] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3037.160880] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3037.176675] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.200565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3042.304337] LustreError: 103712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.327833] LustreError: 103712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3042.538983] Lustre: server umount lustre-MDT0001 complete [ 3054.578975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.703479] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3054.705881] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3054.760229] Lustre: 103704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3055.443958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3055.448223] Lustre: Skipped 1 previous similar message [ 3056.451035] LustreError: 107601:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3056.466515] LustreError: 107601:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3056.484720] LustreError: 103709:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3056.648097] LustreError: 107615:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.147@tcp: cfg command failed: rc = -17 [ 3056.660749] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3056.660749] [ 3060.725300] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3060.960373] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3073.992746] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:11:27 (1788585087) [ 3098.363307] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3125.677955] Lustre: 109669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3157.729737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3161.687793] Lustre: 110804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3250.146212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.148150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3250.165754] Lustre: Skipped 4 previous similar messages [ 3250.178039] Lustre: Skipped 7 previous similar messages [ 3256.130319] Lustre: server umount lustre-MDT0000 complete [ 3260.384493] LustreError: 107549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.410681] LustreError: 107549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 3261.114083] LustreError: 103698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585276 with bad export cookie 15684777630858058038 [ 3261.117615] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3261.133927] LustreError: 103698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3261.479360] Lustre: server umount lustre-MDT0001 complete [ 3276.457244] Lustre: server umount lustre-OST0000 complete [ 3291.637538] Lustre: server umount lustre-OST0001 complete [ 3298.188866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3308.233252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3319.529414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3331.597367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3355.868118] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3369.853408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.879294] Lustre: lustre-MDT0000: reset Object Index mappings [ 3370.452534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.584179] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3387.683634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.717131] Lustre: lustre-MDT0001: reset Object Index mappings [ 3394.676406] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3398.040961] Lustre: 115906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3405.541610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3412.832412] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3413.489483] LustreError: 116260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.511078] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3413.525657] LustreError: 116260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3415.539440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3422.001057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3427.843736] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3427.846709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3429.325459] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3437.756709] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3442.316614] Lustre: 117783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3498.533740] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:18:32 (1788585512) [ 3525.101313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.118037] Lustre: Skipped 2 previous similar messages [ 3525.134434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3525.148947] Lustre: Skipped 7 previous similar messages [ 3538.399168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3538.726255] Lustre: server umount lustre-MDT0000 complete [ 3542.919492] LustreError: 114740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585558 with bad export cookie 15684777630858104889 [ 3542.933177] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3542.936144] LustreError: 114740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3545.569313] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.574976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3545.587409] Lustre: Skipped 2 previous similar messages [ 3545.609238] Lustre: Skipped 9 previous similar messages [ 3549.562421] Lustre: server umount lustre-MDT0001 complete [ 3560.049483] Lustre: server umount lustre-OST0000 complete [ 3570.546067] Lustre: server umount lustre-OST0001 complete [ 3578.767791] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 3588.446300] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3629.774129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.111232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3630.130851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3630.296145] Lustre: lustre-MDT0000: new disk, initializing [ 3630.364378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.367222] Lustre: Skipped 3 previous similar messages [ 3630.381458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3635.910151] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3644.030931] Lustre: server umount lustre-MDT0000 complete [ 3654.832309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.381148] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3655.417081] Lustre: 122069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3655.432905] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3661.362631] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3666.813163] Lustre: server umount lustre-MDT0000 complete [ 3677.284480] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:21:30 (1788585690) [ 3679.097452] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3681.443797] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:21:35 (1788585695) [ 3703.360815] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3713.699770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.360274] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3727.448092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.595781] Lustre: 124142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3727.672850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3727.679081] Lustre: Skipped 1 previous similar message [ 3727.784075] Lustre: lustre-MDT0001: new disk, initializing [ 3728.026992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3728.046638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3733.617745] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3739.174267] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3746.987051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3747.325595] Lustre: lustre-OST0000: new disk, initializing [ 3747.332934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3747.353741] Lustre: 125718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3748.666664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3748.686051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3748.868137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3755.116383] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3766.880080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3767.167704] Lustre: lustre-OST0001: new disk, initializing [ 3767.175244] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3767.185944] Lustre: 126590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3768.672293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3768.702182] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3768.833732] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3773.851600] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3787.345313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3792.234710] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4242.862557] Lustre: 133027:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4242.869508] Lustre: 133027:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4242.876985] Lustre: 133027:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4242.887819] Lustre: 133027:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4242.906372] Lustre: 133027:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4242.918030] Lustre: 133027:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4552.818839] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:36:05 (1788586565) [ 4572.640290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4572.642954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4572.684955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4572.702405] Lustre: Skipped 5 previous similar messages [ 4575.572298] Lustre: server umount lustre-MDT0000 complete [ 4577.768554] LustreError: 127233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.807512] LustreError: 127233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 4580.036113] LustreError: 133077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586595 with bad export cookie 15684777630858173608 [ 4580.055742] LustreError: 133077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4580.057591] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4582.909274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.927174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4582.928471] Lustre: Skipped 4 previous similar messages [ 4582.943801] Lustre: Skipped 1 previous similar message [ 4586.630975] Lustre: server umount lustre-MDT0001 complete [ 4597.698892] Lustre: server umount lustre-OST0000 complete [ 4608.012303] Lustre: server umount lustre-OST0001 complete [ 4616.094391] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 4628.616830] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 4673.092521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.395286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4673.431300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4673.529903] Lustre: lustre-MDT0000: new disk, initializing [ 4673.599615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.605941] Lustre: Skipped 5 previous similar messages [ 4673.619891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4679.089210] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4691.880216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4692.019029] Lustre: 136088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4692.102197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4692.111349] Lustre: Skipped 1 previous similar message [ 4692.211737] Lustre: lustre-MDT0001: new disk, initializing [ 4692.384378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4692.402351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4697.841560] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4706.946341] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4713.393803] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4721.257814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4721.592672] Lustre: lustre-OST0000: new disk, initializing [ 4721.598957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4721.605966] Lustre: 138054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4723.709252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4723.731725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4723.882550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4729.181873] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4739.035211] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4748.853803] Lustre: lustre-MDD0000: changelog on [ 4751.318121] Lustre: lustre-MDD0001: changelog on [ 6760.533764] Lustre: lustre-MDD0001: changelog off [ 6762.731770] Lustre: 144029:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6763.694471] Lustre: lustre-MDD0000: changelog off [ 6770.160705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.171491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6770.181603] Lustre: Skipped 1 previous similar message [ 6770.195409] Lustre: Skipped 3 previous similar messages [ 6775.265329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6775.296342] Lustre: Skipped 1 previous similar message [ 6776.489606] Lustre: server umount lustre-OST0000 complete [ 6781.410537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6781.418226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.432710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6785.508216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6785.508993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.035850] Lustre: server umount lustre-MDT0000 complete [ 6790.624206] LustreError: 136099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6790.650107] LustreError: 136099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6791.922581] LustreError: 138065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588807 with bad export cookie 15684777630859005320 [ 6791.924729] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6791.932917] LustreError: 138065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6792.478480] Lustre: server umount lustre-MDT0001 complete [ 6806.187704] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 6809.767060] Key type lgssc unregistered [ 6810.139139] LNet: 147219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6810.151375] LNetError: 147219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6811.173191] LNet: Removed LNI 192.168.202.147@tcp [ 6812.356216] Key type .llcrypt unregistered [ 6812.358528] Key type ._llcrypt unregistered [ 6835.476209] Key type ._llcrypt registered [ 6835.477785] Key type .llcrypt registered [ 6835.579786] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 6852.531558] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 6853.855226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6853.908826] alg: No test for adler32 (adler32-zlib) [ 6855.110922] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 6855.452435] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 6857.114016] Key type lgssc registered [ 6858.749780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6902.779069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6902.857758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.241210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6904.299770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6904.414446] Lustre: lustre-MDT0000: new disk, initializing [ 6904.573834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6904.599731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6909.494085] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6922.751608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.867576] Lustre: 150420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6922.931037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6922.940694] Lustre: Skipped 1 previous similar message [ 6923.057197] Lustre: lustre-MDT0001: new disk, initializing [ 6923.254267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6923.318175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6923.341714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6929.100753] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6939.930775] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6946.009415] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6954.313314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6954.578174] Lustre: lustre-OST0000: new disk, initializing [ 6954.581627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6954.586828] Lustre: 152390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6954.673726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.626913] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6962.270706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6962.282562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6962.376418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6970.851482] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6976.587875] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_module obdecho/obdecho [ 6978.612687] LustreError: 153396:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6978.618474] LustreError: 153396:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6980.667563] LustreError: 153492:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6993.379828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.382815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.404054] Lustre: Skipped 2 previous similar messages [ 6993.412243] Lustre: Skipped 1 previous similar message [ 6998.072668] Lustre: server umount lustre-MDT0000 complete [ 7002.093487] LustreError: 152388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589017 with bad export cookie 3547687707502175588 [ 7002.099236] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7002.110027] LustreError: 152388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7002.618335] Lustre: server umount lustre-MDT0001 complete [ 7017.477833] Lustre: server umount lustre-OST0000 complete [ 7041.450548] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 7053.525635] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 7098.447894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7098.795898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7098.830858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7098.993450] Lustre: lustre-MDT0000: new disk, initializing [ 7099.150124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7099.185502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7105.049213] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7117.322534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.505089] Lustre: 157210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7117.627283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7117.632581] Lustre: Skipped 1 previous similar message [ 7117.777895] Lustre: lustre-MDT0001: new disk, initializing [ 7117.858681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7117.923061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7117.941339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7123.708320] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7134.511649] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7141.214695] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7148.431048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.894846] Lustre: lustre-OST0000: new disk, initializing [ 7148.898873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7148.910852] Lustre: 159180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7149.107100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7155.673969] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7157.848936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7157.855811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7157.947923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7167.847600] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7173.962995] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7174.160608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7179.082924] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7179.363681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7183.844964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7183.846638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7183.870329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7183.882910] Lustre: Skipped 1 previous similar message [ 7188.961809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7188.974815] Lustre: Skipped 1 previous similar message [ 7194.102762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7194.105517] Lustre: Skipped 1 previous similar message [ 7196.127117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7196.406894] Lustre: server umount lustre-OST0000 complete [ 7204.321448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7204.339157] Lustre: Skipped 1 previous similar message [ 7204.342814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7207.395709] Lustre: server umount lustre-MDT0000 complete [ 7209.441771] LustreError: 157223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7209.456035] LustreError: 157223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7211.687936] LustreError: 157204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589226 with bad export cookie 3547687707502176547 [ 7211.692204] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7211.694434] LustreError: 157204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7212.081637] Lustre: server umount lustre-MDT0001 complete [ 7216.521507] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:20:30 (1788589230) [ 7218.259295] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7220.528303] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:20:34 (1788589234) [ 7229.486361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.151085] LustreError: 161597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7230.269657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7235.556096] LustreError: 161598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7235.889201] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7240.676213] LustreError: 161597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.800787] LustreError: 161598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.134869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.474187] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7261.143694] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7267.949945] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7275.889061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.345308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.355025] Lustre: Skipped 1 previous similar message [ 7283.242970] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7291.498139] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7310.314852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.322699] Lustre: Skipped 1 previous similar message [ 7310.326647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.336873] Lustre: Skipped 2 previous similar messages [ 7314.381874] Lustre: server umount lustre-MDT0000 complete [ 7315.437914] LustreError: 161598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.464736] LustreError: 161598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7319.235462] LustreError: 161577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589334 with bad export cookie 3547687707502177429 [ 7319.248844] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7319.252829] LustreError: 161577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7320.551076] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.574702] Lustre: Skipped 1 previous similar message [ 7326.071955] Lustre: server umount lustre-MDT0001 complete [ 7334.001103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7347.946651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7348.204043] LustreError: 165465:0:(llog.c:1655:llog_backup()) MGC192.168.202.147@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7348.220540] Lustre: 165465:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.147@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7349.216465] LustreError: 165470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7349.232920] LustreError: 165470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7384.063665] LustreError: 165471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.088108] LustreError: 165471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7409.119852] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7409.123515] Lustre: 165465:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.147@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7449.608342] LustreError: 165491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.636608] LustreError: 165491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7450.151535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7455.216675] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7457.299168] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7468.279321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.819814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7496.182160] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7497.520606] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7506.201240] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7508.046728] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7514.714353] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7516.458988] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7542.241194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.246476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7542.288285] Lustre: Skipped 2 previous similar messages [ 7542.296344] Lustre: Skipped 6 previous similar messages [ 7545.212937] Lustre: server umount lustre-MDT0000 complete [ 7550.057226] LustreError: 161578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589565 with bad export cookie 3547687707502183918 [ 7550.058328] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.071479] LustreError: 161578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7550.575495] Lustre: server umount lustre-MDT0001 complete [ 7565.969833] Lustre: server umount lustre-OST0000 complete [ 7591.956603] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 7604.677596] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 7645.705989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.042372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7646.075414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7646.203372] Lustre: lustre-MDT0000: new disk, initializing [ 7646.277031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.283204] Lustre: Skipped 1 previous similar message [ 7646.301808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7651.726733] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7663.807189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.998079] Lustre: 170582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7664.375170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7668.722372] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7678.762124] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7686.117108] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7694.421358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7694.859200] Lustre: lustre-OST0000: new disk, initializing [ 7694.862032] Lustre: Skipped 1 previous similar message [ 7694.865636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7694.872492] Lustre: Skipped 2 previous similar messages [ 7694.878455] Lustre: 172553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7702.364775] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7704.205136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7704.222533] Lustre: Skipped 1 previous similar message [ 7704.238284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7704.379429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7712.828706] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7718.476697] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7718.693380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7723.635903] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7723.812809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7730.160340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7730.164070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7730.166665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7730.178578] Lustre: Skipped 1 previous similar message [ 7730.185140] Lustre: Skipped 2 previous similar messages [ 7740.385790] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7740.969637] Lustre: server umount lustre-OST0000 complete [ 7752.012374] Lustre: server umount lustre-MDT0000 complete [ 7755.745029] LustreError: 170592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7755.776550] LustreError: 170592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7756.337185] LustreError: 170574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589771 with bad export cookie 3547687707502200725 [ 7756.344784] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7756.353789] LustreError: 170574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7756.850344] Lustre: server umount lustre-MDT0001 complete [ 7765.769479] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:29:39 (1788589779) [ 7772.445983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.271306] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7786.383547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.895562] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7798.880408] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7805.494232] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7815.643614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7824.307611] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7831.932517] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7841.341210] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:30:55 (1788589855) [ 7846.881791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7846.887783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.899611] LustreError: Skipped 1 previous similar message [ 7846.916982] Lustre: Skipped 3 previous similar messages [ 7846.918141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7846.933965] Lustre: Skipped 9 previous similar messages [ 7852.691915] Lustre: server umount lustre-OST0000 complete [ 7858.662244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7863.744116] Lustre: server umount lustre-MDT0000 complete [ 7868.120411] LustreError: 174945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589883 with bad export cookie 3547687707502201607 [ 7868.132752] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7868.142097] LustreError: 174945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7868.395355] Lustre: server umount lustre-MDT0001 complete [ 7881.398960] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 7884.219940] Key type lgssc unregistered [ 7884.624716] LNet: 178695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7884.634189] LNetError: 178695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7884.665676] LNet: Removed LNI 192.168.202.147@tcp [ 7885.804628] Key type .llcrypt unregistered [ 7885.806742] Key type ._llcrypt unregistered [ 7914.759299] Key type ._llcrypt registered [ 7914.765981] Key type .llcrypt registered [ 7914.978676] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 7917.051039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7917.134151] alg: No test for adler32 (adler32-zlib) [ 7918.356431] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 7918.812490] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 7920.615239] Key type lgssc registered [ 7921.477098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7929.084708] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7929.097464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7944.673462] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7949.791504] LustreError: 179943:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.147@tcp: failed processing log, type 4: rc = -110 [ 7954.911935] LustreError: 179943:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7954.930210] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7954.942842] LustreError: MGC192.168.202.147@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7954.955143] LustreError: 179943:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7954.983887] LustreError: 179943:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7954.990564] LustreError: 179943:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7955.091364] Lustre: server umount lustre-OST0000 complete [ 7955.094575] LustreError: 179943:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7965.550495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.288487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7973.590387] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7977.447702] LustreError: 180443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7977.486902] LustreError: 180443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7982.568399] LustreError: 180444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7982.746058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7983.187865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7988.023088] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7995.431075] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8001.747190] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8011.807505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8012.112372] LustreError: 182259:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8012.121327] Lustre: 182259:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.147@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8012.333769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.566203] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8028.019045] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8033.775181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.779929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8033.803660] Lustre: Skipped 2 previous similar messages [ 8033.811163] Lustre: Skipped 1 previous similar message [ 8036.523128] Lustre: server umount lustre-MDT0000 complete [ 8038.883741] LustreError: 181430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8038.903457] LustreError: 181430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8039.799600] LustreError: 183019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590055 with bad export cookie 15279440364490740343 [ 8039.804655] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8039.813394] LustreError: 183019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8040.080026] Lustre: server umount lustre-MDT0001 complete [ 8053.786185] Lustre: server umount lustre-OST0000 complete [ 8063.565555] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 8073.567747] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8108.445706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.792532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8108.848666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8109.034543] Lustre: lustre-MDT0000: new disk, initializing [ 8109.174390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8109.195319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8114.426943] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8125.280582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8125.396805] Lustre: 185784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8125.444559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8125.449216] Lustre: Skipped 1 previous similar message [ 8125.509821] Lustre: lustre-MDT0001: new disk, initializing [ 8125.628198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8125.673362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8125.691058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8130.590114] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8139.010356] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8144.926142] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8151.160751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8151.498149] Lustre: lustre-OST0000: new disk, initializing [ 8151.503716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8151.517310] Lustre: 187751:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8151.677756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8157.412396] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8161.335795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8161.353138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8161.474664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8166.188512] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8170.385081] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8170.580546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8174.679569] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8174.876520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8177.124599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8177.135451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.138835] LustreError: Skipped 1 previous similar message [ 8177.146659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8177.156383] Lustre: Skipped 1 previous similar message [ 8177.176829] Lustre: Skipped 2 previous similar messages [ 8182.248531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.260249] Lustre: Skipped 1 previous similar message [ 8182.832772] Lustre: server umount lustre-OST0000 complete [ 8187.362725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.370099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8187.396124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8192.483221] LustreError: 187255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8192.626448] Lustre: server umount lustre-MDT0000 complete [ 8195.858655] LustreError: 185776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590211 with bad export cookie 15279440364490741064 [ 8195.861109] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8195.867572] LustreError: 185776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8196.176963] Lustre: server umount lustre-MDT0001 complete [ 8204.415529] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:36:58 (1788590218) [ 8220.188212] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 8223.017327] Key type lgssc unregistered [ 8223.303824] LNet: 190693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8223.311233] LNetError: 190693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8223.328621] LNet: Removed LNI 192.168.202.147@tcp [ 8224.403977] Key type .llcrypt unregistered [ 8224.406837] Key type ._llcrypt unregistered [ 8240.729536] Key type ._llcrypt registered [ 8240.733778] Key type .llcrypt registered [ 8240.834853] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8241.883664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8242.004290] alg: No test for adler32 (adler32-zlib) [ 8242.995837] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 8243.174952] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 8244.783193] Key type lgssc registered [ 8245.822261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8254.202220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8254.230772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.857959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8261.225658] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8266.210588] LustreError: 191910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8266.241124] LustreError: 191910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8270.244437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8270.677647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8275.135374] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8282.159849] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8287.327942] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8295.529617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.782893] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8295.791125] LustreError: 193773:0:(llog.c:1696:llog_backup()) MGC192.168.202.147@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8295.805900] Lustre: 193773:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.147@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8295.984767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8303.149104] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8312.083349] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8315.946638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.966136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8320.995147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8320.999549] Lustre: Skipped 2 previous similar messages [ 8321.260922] Lustre: server umount lustre-MDT0000 complete [ 8325.992534] LustreError: 191891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590341 with bad export cookie 11187810181105240632 [ 8326.017770] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8326.126180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8326.133652] LustreError: 191905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8326.150423] Lustre: Skipped 2 previous similar messages [ 8326.152442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8326.152450] Lustre: Skipped 2 previous similar messages [ 8326.279630] LustreError: 192639:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c94e35f6 x1875472818400256/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8326.520530] Lustre: server umount lustre-MDT0001 complete [ 8333.671384] Lustre: server umount lustre-OST0000 complete [ 8345.100770] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 8357.627674] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8401.204280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.469818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8401.527863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8401.683762] Lustre: lustre-MDT0000: new disk, initializing [ 8401.787393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8401.828059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8406.807363] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8419.626817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8419.894486] Lustre: 197295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8419.955248] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8419.957432] Lustre: Skipped 1 previous similar message [ 8419.960467] LustreError: 198080:0:(llog.c:1696:llog_backup()) MGC192.168.202.147@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8419.971698] LustreError: 198080:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8419.977502] Lustre: 198080:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.147@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8420.007996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8420.010824] Lustre: Skipped 1 previous similar message [ 8420.088800] Lustre: lustre-MDT0001: new disk, initializing [ 8420.198499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8420.238312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8420.256458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8427.072242] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8439.819473] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8446.450473] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8455.020905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8455.337391] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8455.341656] Lustre: Skipped 1 previous similar message [ 8455.346451] LustreError: 199243:0:(llog.c:1696:llog_backup()) MGC192.168.202.147@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8455.367985] LustreError: 199243:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8455.376144] Lustre: 199243:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.147@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8455.382474] Lustre: 199243:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8455.417547] Lustre: lustre-OST0000: new disk, initializing [ 8455.425618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8455.443124] Lustre: 199266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8455.583608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8460.941302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8460.958133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8461.071475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8462.829386] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8471.778476] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8476.466490] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8476.692192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8481.411560] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8481.647147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8486.880959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8486.886699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8486.893944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8486.919573] Lustre: Skipped 1 previous similar message [ 8492.001400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8492.015083] Lustre: Skipped 2 previous similar messages [ 8498.146331] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8498.294338] Lustre: server umount lustre-OST0000 complete [ 8507.360819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.361819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8507.382935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8507.382945] Lustre: Skipped 2 previous similar messages [ 8507.400379] LustreError: Skipped 1 previous similar message [ 8508.671879] Lustre: server umount lustre-MDT0000 complete [ 8512.473365] LustreError: 197289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590527 with bad export cookie 11187810181105241346 [ 8512.494469] LustreError: 197289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8512.499698] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8512.989376] Lustre: server umount lustre-MDT0001 complete [ 8521.757099] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:42:15 (1788590535) [ 8540.027773] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8549.995945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.438513] LustreError: 202395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.495784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8555.499748] LustreError: 202396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8555.816996] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8560.629074] LustreError: 202395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8564.334730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.608333] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8571.348119] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8577.929115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8584.258207] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8592.469978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8592.658946] Lustre: lustre-OST0001: new disk, initializing [ 8592.665785] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8592.675246] Lustre: 204704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8592.788042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8592.791030] Lustre: Skipped 2 previous similar messages [ 8599.038093] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8600.724126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8600.734342] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8600.842210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8609.705519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8621.085842] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8623.638435] Lustre: DEBUG MARKER: ADD OST3 [ 8630.200750] loop: module loaded [ 8630.331456] loop0: detected capacity change from 0 to 204800000 [ 8630.691619] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8630.708980] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8630.732614] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8638.162488] Lustre: DEBUG MARKER: STOP OST3 [ 8638.247445] loop0: detected capacity change from 0 to 204800000 [ 8644.762747] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8644.993214] Lustre: lustre-OST0003: new disk, initializing [ 8645.000286] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8645.007981] Lustre: 206782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8646.873195] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8647.036159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8647.047567] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8647.110495] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8648.211600] Lustre: Failing over lustre-OST0003 [ 8648.481230] Lustre: server umount lustre-OST0003 complete [ 8648.927886] LustreError: 206904:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8651.806792] LustreError: 203882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8652.269587] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.272119] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.290358] Lustre: Skipped 1 previous similar message [ 8654.396335] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8664.571077] Lustre: DEBUG MARKER: START OST3 again [ 8671.981860] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8672.234482] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8672.251591] Lustre: Skipped 1 previous similar message [ 8672.273504] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8673.395771] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8674.290251] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8674.299800] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8679.084207] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8699.387551] Lustre: Failing over lustre-OST0003 [ 8699.476309] Lustre: server umount lustre-OST0003 complete [ 8699.873827] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8699.883551] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.886778] LustreError: Skipped 1 previous similar message [ 8699.889806] LustreError: 203882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.889819] LustreError: 203882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8699.973201] Lustre: Skipped 2 previous similar messages [ 8718.310683] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.325889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8718.325960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8718.340502] Lustre: Skipped 3 previous similar messages [ 8718.366428] Lustre: Skipped 4 previous similar messages [ 8723.520765] Lustre: server umount lustre-MDT0000 complete [ 8727.495748] LustreError: 202377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590742 with bad export cookie 11187810181105242235 [ 8727.521916] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8727.525857] LustreError: 202377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8732.578552] LustreError: 204509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8732.601302] LustreError: 204509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8734.136259] Lustre: server umount lustre-MDT0001 complete [ 8738.366414] Lustre: server umount lustre-OST0000 complete [ 8742.424706] Lustre: server umount lustre-OST0001 complete [ 8749.920649] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 8759.238721] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8799.385048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.643261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8799.677735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8799.772749] Lustre: lustre-MDT0000: new disk, initializing [ 8799.869190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8805.142308] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8815.229022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8815.388643] Lustre: 212316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8815.556560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8815.561387] Lustre: Skipped 1 previous similar message [ 8815.613471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8820.119804] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8829.767830] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8835.492559] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8842.584951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8842.904822] Lustre: lustre-OST0000: new disk, initializing [ 8842.907942] Lustre: Skipped 1 previous similar message [ 8842.912353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8842.916497] Lustre: Skipped 2 previous similar messages [ 8842.921207] Lustre: 214284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8844.737903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8844.745486] Lustre: Skipped 1 previous similar message [ 8844.749328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8844.881159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8848.939219] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8858.407235] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8862.939952] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8863.209046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8867.627685] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8867.824841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8870.374578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8870.375744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8870.393527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8870.404163] Lustre: Skipped 3 previous similar messages [ 8870.415387] Lustre: Skipped 5 previous similar messages [ 8876.258846] Lustre: server umount lustre-OST0000 complete [ 8886.207612] Lustre: server umount lustre-MDT0000 complete [ 8889.924326] LustreError: 212307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590905 with bad export cookie 11187810181105244531 [ 8889.941771] LustreError: 212307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8889.945485] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8890.250086] Lustre: server umount lustre-MDT0001 complete [ 8893.756224] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:48:27 (1788590907) [ 8908.481400] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 8916.892291] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8952.520813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8952.822379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8952.880949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8952.982823] Lustre: lustre-MDT0000: new disk, initializing [ 8953.095669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8957.616493] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8968.226717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.312973] Lustre: 218893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8968.509679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8972.334685] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8981.980164] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8988.081746] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8994.778283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8995.057676] Lustre: 220862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8996.883765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9001.267965] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9010.573474] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9014.721454] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9014.956810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9020.060773] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9020.294808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9027.556092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.570377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9027.572241] Lustre: Skipped 2 previous similar messages [ 9027.573776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9027.573782] Lustre: Skipped 6 previous similar messages [ 9028.755714] Lustre: server umount lustre-OST0000 complete [ 9038.839279] Lustre: server umount lustre-MDT0000 complete [ 9042.636446] LustreError: 218886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591057 with bad export cookie 11187810181105245413 [ 9042.637635] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9042.650879] LustreError: 218886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9042.929897] Lustre: server umount lustre-MDT0001 complete [ 9051.950648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9052.564293] LustreError: 223081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9052.581883] LustreError: 223081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9058.155480] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9068.792433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.105618] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9083.317516] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9090.411259] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9097.239316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9097.624258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9097.627527] Lustre: Skipped 6 previous similar messages [ 9105.486520] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9114.028095] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9137.634161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9137.642612] LustreError: Skipped 1 previous similar message [ 9141.240940] Lustre: server umount lustre-OST0000 complete [ 9151.344855] Lustre: server umount lustre-MDT0000 complete [ 9154.916666] LustreError: 223063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591170 with bad export cookie 11187810181105246295 [ 9154.920923] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9154.932065] LustreError: 223063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9155.245153] Lustre: server umount lustre-MDT0001 complete [ 9254.916893] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 9258.704541] Key type lgssc unregistered [ 9259.009283] LNet: 226771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9259.020926] LNetError: 226771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9259.055540] LNet: Removed LNI 192.168.202.147@tcp [ 9260.583313] Key type .llcrypt unregistered [ 9260.588190] Key type ._llcrypt unregistered [ 9294.154228] Key type ._llcrypt registered [ 9294.158538] Key type .llcrypt registered [ 9294.268043] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 9310.280801] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 9311.762862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9311.863344] alg: No test for adler32 (adler32-zlib) [ 9313.148412] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 9313.498916] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 9315.175607] Key type lgssc registered [ 9316.687803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9353.814085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9353.856637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9355.084324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9355.103073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9355.199377] Lustre: lustre-MDT0000: new disk, initializing [ 9355.332604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9355.367799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9359.801811] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9372.856806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.975563] Lustre: 230122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9373.022586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9373.026476] Lustre: Skipped 1 previous similar message [ 9373.094770] Lustre: lustre-MDT0001: new disk, initializing [ 9373.151795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9373.190844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9373.201434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9377.607262] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9387.881757] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9395.941634] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9403.300038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9403.703577] Lustre: lustre-OST0000: new disk, initializing [ 9403.716316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9403.736533] Lustre: 232095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9403.845800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9410.436547] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9411.684898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9411.698447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9411.859277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9420.257033] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9425.366863] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9425.589710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9432.162747] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9432.402539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9437.665250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9437.668731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9437.669035] LustreError: Skipped 1 previous similar message [ 9437.676336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9437.679704] Lustre: Skipped 1 previous similar message [ 9441.238850] Lustre: server umount lustre-OST0000 complete [ 9447.911401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9447.913422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9447.953186] Lustre: Skipped 1 previous similar message [ 9447.961831] Lustre: Skipped 1 previous similar message [ 9452.462732] Lustre: server umount lustre-MDT0000 complete [ 9456.853978] LustreError: 230920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591472 with bad export cookie 2184399070629904921 [ 9456.858588] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9456.869136] LustreError: 230920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9457.149362] Lustre: server umount lustre-MDT0001 complete [ 9476.796691] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 9486.660133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.035097] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.054598] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9487.080371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.218424] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9492.460277] LustreError: 235027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.568832] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9500.160733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9500.445574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9503.997703] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9507.152268] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9514.347539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9514.745784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9521.251233] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9529.621163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9529.765838] Lustre: lustre-OST0001: new disk, initializing [ 9529.769601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9529.778322] Lustre: 237333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9536.002961] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9536.634436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9536.652019] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9536.774798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9547.790687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9557.916221] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9570.492384] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:59:44 (1788591584) [ 9572.832803] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.843072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9572.855735] Lustre: Skipped 1 previous similar message [ 9572.865647] Lustre: Skipped 2 previous similar messages [ 9577.447214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9577.456437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9577.470729] Lustre: Skipped 1 previous similar message [ 9577.478478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9579.194696] Lustre: server umount lustre-MDT0000 complete [ 9583.109434] LustreError: 235021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.141264] LustreError: 235021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9583.272353] LustreError: 236160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591598 with bad export cookie 2184399070629905803 [ 9583.279240] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9583.283143] LustreError: 236160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9583.840370] Lustre: server umount lustre-MDT0001 complete [ 9597.589266] Lustre: server umount lustre-OST0000 complete [ 9612.455130] Lustre: server umount lustre-OST0001 complete [ 9619.476718] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 9630.275941] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 9669.451901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9669.697788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9669.727934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9669.802610] Lustre: lustre-MDT0000: new disk, initializing [ 9669.888929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9669.897799] Lustre: Skipped 1 previous similar message [ 9669.918727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9674.691935] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9687.257852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.497565] Lustre: 241307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9687.632754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9687.642737] Lustre: Skipped 1 previous similar message [ 9687.780565] Lustre: lustre-MDT0001: new disk, initializing [ 9688.008363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9688.035189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9693.796925] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9706.344122] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9713.683960] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9720.545783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9720.751192] Lustre: lustre-OST0000: new disk, initializing [ 9720.754214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9720.762229] Lustre: 243278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9720.817865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9720.820339] Lustre: Skipped 1 previous similar message [ 9722.720236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9722.736509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9722.934351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9727.388899] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9737.619028] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9742.982640] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9743.173272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9747.848578] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9748.021631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9753.570381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9753.571310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.578218] LustreError: Skipped 1 previous similar message [ 9753.581903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9753.581914] Lustre: Skipped 3 previous similar messages [ 9753.628518] Lustre: Skipped 1 previous similar message [ 9763.826058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9763.834743] Lustre: Skipped 4 previous similar messages [ 9764.322372] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9764.517426] Lustre: server umount lustre-OST0000 complete [ 9769.961274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9769.979621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9775.951698] Lustre: server umount lustre-MDT0000 complete [ 9779.168937] LustreError: 241318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9779.190797] LustreError: 241318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9779.826255] LustreError: 243277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591795 with bad export cookie 2184399070629907931 [ 9779.831878] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9779.840159] LustreError: 243277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9780.141837] Lustre: server umount lustre-MDT0001 complete [ 9788.390892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.955161] LustreError: 245495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9789.067095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.783257] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9803.077169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.407420] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9817.796520] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9825.473504] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9834.353062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9842.217588] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9849.911451] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9996.771644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9996.775705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.781259] Lustre: Skipped 1 previous similar message [ 9996.783736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9996.785985] Lustre: Skipped 3 previous similar messages [10001.280600] Lustre: server umount lustre-OST0000 complete [10019.296801] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10019.666984] Lustre: server umount lustre-MDT0000 complete [10023.400555] LustreError: 245490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.441974] LustreError: 245490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10025.609953] LustreError: 246232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592040 with bad export cookie 2184399070629908820 [10025.620943] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10025.638887] LustreError: 246232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10026.540308] Lustre: server umount lustre-MDT0001 complete [10119.000591] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [10122.394751] Key type lgssc unregistered [10122.855348] LNet: 249103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10122.867827] LNetError: 249103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10122.886534] LNet: Removed LNI 192.168.202.147@tcp [10123.867781] Key type .llcrypt unregistered [10123.874342] Key type ._llcrypt unregistered [10151.327819] Key type ._llcrypt registered [10151.329436] Key type .llcrypt registered [10151.427389] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [10169.008903] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10170.262178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10170.321567] alg: No test for adler32 (adler32-zlib) [10171.491355] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [10171.783962] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [10173.431258] Key type lgssc registered [10174.503491] Lustre: Echo OBD driver; http://www.lustre.org/ [10221.176990] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10232.030098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10232.040689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10233.361481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10233.416718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10233.505778] Lustre: lustre-MDT0000: new disk, initializing [10233.618328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10233.653303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10238.867674] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10250.425498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10250.542648] Lustre: 253170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10250.582789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10250.585828] Lustre: Skipped 1 previous similar message [10250.653667] Lustre: lustre-MDT0001: new disk, initializing [10250.792651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10250.883919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10250.897402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10254.998852] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10260.289102] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10267.845679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10268.096302] Lustre: lustre-OST0000: new disk, initializing [10268.098716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10268.102244] Lustre: 254800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10268.183321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10274.092859] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10276.481537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10276.501583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10276.676031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10285.993753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.214279] Lustre: lustre-OST0001: new disk, initializing [10286.218529] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10286.226916] Lustre: 255670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10286.452424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10292.678246] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10295.503128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10295.511017] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10295.642072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10304.151579] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10312.915606] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10326.168621] Lustre: server umount lustre-MDT0000 complete [10326.510983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10326.544964] Lustre: Skipped 2 previous similar messages [10329.672055] LustreError: 253160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592344 with bad export cookie 1534721745419931431 [10329.684803] LustreError: 253160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10329.698384] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10330.061730] Lustre: server umount lustre-MDT0001 complete [10344.620460] Lustre: server umount lustre-OST0000 complete [10347.488556] Lustre: 250674:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788592346/real 1788592346] req@ffff8a363df52a00 x1875474841151616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788592362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10347.523903] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10347.545552] Lustre: Skipped 1 previous similar message [10348.436335] Lustre: server umount lustre-OST0001 complete [10362.296204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10372.670986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10381.208714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10392.734610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10414.238965] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10423.792720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10423.816044] Lustre: lustre-MDT0000: reset Object Index mappings [10424.129812] LustreError: 260682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.157772] LustreError: 260682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10424.235968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10428.404144] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10429.412148] LustreError: 260683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10434.540181] LustreError: 260682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.778152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10436.793330] Lustre: lustre-MDT0001: reset Object Index mappings [10437.145731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10441.465366] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10444.222584] Lustre: 261824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10450.965546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10457.274350] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10462.565626] LustreError: 262177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10465.307950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10465.498566] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10465.502487] Lustre: Skipped 1 previous similar message [10469.640402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10469.645112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10471.395643] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10478.695148] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10482.006680] Lustre: 263693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10488.295409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10488.305510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.317133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10489.388494] Lustre: server umount lustre-MDT0000 complete [10490.345969] LustreError: 260683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.357623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.378908] Lustre: Skipped 2 previous similar messages [10492.574071] LustreError: 260663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592507 with bad export cookie 1534721745419938221 [10492.578116] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.583253] LustreError: 260663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10492.825540] Lustre: server umount lustre-MDT0001 complete [10506.848809] Lustre: server umount lustre-OST0000 complete [10520.435404] Lustre: server umount lustre-OST0001 complete [10538.795851] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:15:52 (1788592552) [10553.380216] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [10564.692575] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10599.933401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.125204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10600.145213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10600.213396] Lustre: lustre-MDT0000: new disk, initializing [10600.358455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10600.376619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10604.924912] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10616.210631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.413908] Lustre: 267947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10616.564299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10616.569949] Lustre: Skipped 1 previous similar message [10616.658314] Lustre: lustre-MDT0001: new disk, initializing [10616.812491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10616.834165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10621.695925] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10631.863443] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10638.392814] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10645.039790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10645.420810] Lustre: lustre-OST0000: new disk, initializing [10645.423802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10645.430215] Lustre: 269917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10647.307242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10647.324692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10647.492097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10651.351312] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10661.001472] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10665.829565] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10666.016982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10670.639542] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10670.820216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10673.121405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10673.123781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10673.145301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10673.151229] Lustre: Skipped 1 previous similar message [10678.246860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10678.259927] Lustre: Skipped 1 previous similar message [10679.003451] Lustre: server umount lustre-OST0000 complete [10683.368625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.372701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10683.397219] Lustre: Skipped 2 previous similar messages [10683.400434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.407737] Lustre: Skipped 1 previous similar message [10688.481761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10688.486356] Lustre: Skipped 1 previous similar message [10689.585621] Lustre: server umount lustre-MDT0000 complete [10693.278364] LustreError: 268741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592708 with bad export cookie 1534721745419940307 [10693.283075] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10693.283601] LustreError: 268741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10693.666707] Lustre: server umount lustre-MDT0001 complete [10715.374370] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10727.680744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10728.274577] LustreError: 272853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10728.306606] LustreError: 272853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10728.425909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10728.446305] Lustre: Skipped 2 previous similar messages [10734.390100] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10743.722746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10749.468569] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10752.925886] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10761.315585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10767.446121] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10776.312832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10776.483720] Lustre: lustre-OST0001: new disk, initializing [10776.489693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10776.511766] Lustre: 275163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10777.740750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10777.751964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10777.900505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10783.999540] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10793.774679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10797.159579] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10802.817034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10802.865399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10810.854392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10810.860927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10810.886305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10815.779849] Lustre: server umount lustre-MDT0000 complete [10819.046866] LustreError: 275753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10819.082928] LustreError: 275753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10819.666969] LustreError: 272835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592834 with bad export cookie 1534721745419941196 [10819.676357] LustreError: 272835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10819.680351] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.135541] Lustre: server umount lustre-MDT0001 complete [10833.555138] Lustre: server umount lustre-OST0000 complete [10847.870946] Lustre: server umount lustre-OST0001 complete [10864.410912] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10875.321822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10875.782224] LustreError: 278220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.828985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10875.833625] Lustre: Skipped 3 previous similar messages [10882.297451] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10890.977204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10891.234846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10891.252226] Lustre: Skipped 4 previous similar messages [10896.410358] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10899.778187] Lustre: 279360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10906.133741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10913.334348] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10922.009217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10923.018373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10928.646878] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10930.238246] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10938.770952] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10942.884620] Lustre: 281236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10949.440618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10949.514678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10958.812510] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:22:52 (1788592972) [10973.664558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10973.668266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10973.682767] Lustre: Skipped 3 previous similar messages [10973.688651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10980.125908] Lustre: server umount lustre-MDT0000 complete [10984.405638] LustreError: 278201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592999 with bad export cookie 1534721745419947958 [10984.406262] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10984.421951] LustreError: 278201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10984.941367] LustreError: 278215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.972778] LustreError: 278215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10991.597131] Lustre: server umount lustre-MDT0001 complete [11002.189139] Lustre: server umount lustre-OST0000 complete [11006.284793] Lustre: server umount lustre-OST0001 complete [11012.801720] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [11023.226783] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [11069.186728] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [11078.737064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11078.993918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11079.026218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11079.081718] Lustre: lustre-MDT0000: new disk, initializing [11079.162647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11083.927546] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11094.805652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11094.914470] Lustre: 285477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11095.362588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11100.716716] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11113.346707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11113.740129] Lustre: 287111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11115.149149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11120.760967] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11130.508045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11130.678690] Lustre: 287982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11132.263127] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11132.276768] Lustre: Skipped 1 previous similar message [11132.408027] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11136.767463] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11147.387502] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11151.418107] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11151.425310] Lustre: Skipped 1 previous similar message [11167.848476] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:26:21 (1788593181) [11172.327837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11172.340816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11172.351287] Lustre: Skipped 5 previous similar messages [11172.357159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11172.364733] Lustre: Skipped 12 previous similar messages [11172.795696] Lustre: server umount lustre-MDT0000 complete [11173.366220] LustreError: 286288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11173.390271] LustreError: 286288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11177.270780] LustreError: 285470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593192 with bad export cookie 1534721745419955371 [11177.287350] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11177.289588] LustreError: 285470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11177.917568] Lustre: server umount lustre-MDT0001 complete [11193.590578] Lustre: server umount lustre-OST0000 complete [11194.335130] Lustre: 250674:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788593193/real 1788593193] req@ffff8a3507a1f800 x1875474841463936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788593209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11197.694399] Lustre: server umount lustre-OST0001 complete [11206.569155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11206.972975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11206.980491] Lustre: Skipped 7 previous similar messages [11210.927569] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11219.748162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.652533] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11232.317682] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11237.792667] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11244.182129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11251.059930] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11260.691583] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11280.473141] loop1: detected capacity change from 0 to 204800000 [11280.753146] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11280.782545] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11280.812575] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11286.581362] loop1: detected capacity change from 0 to 204800000 [11293.852386] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11294.033674] LustreError: 290215:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13266: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11294.045663] LustreError: 290215:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13266 [11294.050903] LustreError: 290215:0:(mgs_handler.c:517:mgs_target_reg()) fs13266-MDT0001: Error checking target index: rc = -13 [11294.058623] LustreError: fs13266-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11294.069231] LustreError: 293946:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11294.078737] LustreError: 293946:0:(tgt_mount.c:2132:server_put_super()) no obd fs13266-MDT0001 [11294.085519] LustreError: 293946:0:(tgt_mount.c:116:server_deregister_mount()) fs13266-MDT0001 not registered [11294.127170] Lustre: server umount fs13266-MDT0001 complete [11294.133937] LustreError: 293946:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11303.424594] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11303.688634] Lustre: Setting parameter fs13266-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13266-MDT0001 [11303.778671] Lustre: srv-fs13266-MDT0001: No data found on store. Initialize space. [11303.782207] Lustre: Skipped 4 previous similar messages [11303.920445] Lustre: fs13266-MDT0001: new disk, initializing [11303.927664] Lustre: Skipped 3 previous similar messages [11304.091696] Lustre: srv-fs13266-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11304.108898] Lustre: 294412:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13266-MDT0001-mdtlov: can't create new llog: rc = -115 [11304.126889] LustreError: 294412:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13266-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11308.915499] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11316.778286] Lustre: server umount fs13266-MDT0001 complete [11329.528186] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11330.810186] loop1: detected capacity change from 0 to 204800000 [11336.136453] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.360927] Lustre: 295981:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13266-MDT0001-mdtlov: can't create new llog: rc = -115 [11336.375726] LustreError: 295981:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13266-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11340.172749] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11345.511902] Lustre: server umount fs13266-MDT0001 complete [11358.176803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11358.196114] Lustre: Skipped 4 previous similar messages [11358.202209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11362.830962] Lustre: server umount lustre-MDT0000 complete [11373.275750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11373.431635] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11379.024663] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11379.208137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11387.917650] loop1: detected capacity change from 0 to 204800000 [11394.995630] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11395.744326] Lustre: 298429:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13266-MDT0001-mdtlov: can't create new llog: rc = -115 [11395.757888] LustreError: 298429:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13266-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [11399.711079] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11404.985921] Lustre: server umount fs13266-MDT0001 complete [11430.403874] LustreError: 292053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.434573] LustreError: 292053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 114 previous similar messages [11434.979392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11435.005018] LustreError: Skipped 2 previous similar messages [11436.894977] Lustre: server umount lustre-OST0000 complete [11447.441944] Lustre: server umount lustre-MDT0000 complete [11451.591630] LustreError: 290205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593466 with bad export cookie 1534721745419958052 [11451.609240] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11451.870710] Lustre: server umount lustre-MDT0001 complete [11464.848620] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [11468.167982] Key type lgssc unregistered [11468.479658] LNet: 300771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11468.492125] LNetError: 300771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11468.519546] LNet: Removed LNI 192.168.202.147@tcp [11469.347162] Key type .llcrypt unregistered [11469.351287] Key type ._llcrypt unregistered [11489.255839] Key type ._llcrypt registered [11489.261910] Key type .llcrypt registered [11489.368394] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [11490.515686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11490.546592] alg: No test for adler32 (adler32-zlib) [11491.588608] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [11491.799443] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [11493.447217] Key type lgssc registered [11494.494739] Lustre: Echo OBD driver; http://www.lustre.org/ [11502.302265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11502.330990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11503.850224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11509.232456] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11514.337434] LustreError: 302041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11514.355082] LustreError: 302041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11518.262314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11518.774978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11523.949645] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11531.533601] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11537.627865] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11544.849279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11545.243745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11549.358939] LustreError: 303871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.025034] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11554.806000] LustreError: 303873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.822407] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11559.913586] LustreError: 303872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.591729] LustreError: 303872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.618434] LustreError: 303872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11576.845291] LustreError: 303871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11576.887559] LustreError: 303871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11583.938342] loop1: detected capacity change from 0 to 204800000 [11584.261506] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11584.280778] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11584.307424] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11590.961477] loop1: detected capacity change from 0 to 204800000 [11593.708213] LustreError: 303873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.734248] LustreError: 303873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11597.237268] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11597.488137] Lustre: fs26082-OST0000: new disk, initializing [11597.501820] Lustre: srv-fs26082-OST0000: No data found on store. Initialize space. [11597.513398] Lustre: 305778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11597.635281] Lustre: fs26082-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11604.896874] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11612.963995] Lustre: server umount fs26082-OST0000 complete [11627.514370] loop1: detected capacity change from 0 to 204800000 [11628.026110] LustreError: 303871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.037506] LustreError: 303871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11629.540681] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11634.597766] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11634.912593] Lustre: fs26082-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11641.561511] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11644.922068] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11650.816860] loop2: detected capacity change from 0 to 204800000 [11651.087403] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11651.112094] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11651.130263] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11658.306606] loop2: detected capacity change from 0 to 204800000 [11665.973048] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11666.174633] LustreError: fs26082-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11666.194898] LustreError: fs26082-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11666.206889] LustreError: 308819:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11666.216758] LustreError: 308819:0:(tgt_mount.c:2132:server_put_super()) no obd fs26082-OST0001 [11666.223275] LustreError: 308819:0:(tgt_mount.c:116:server_deregister_mount()) fs26082-OST0001 not registered [11666.253182] Lustre: server umount fs26082-OST0001 complete [11666.262192] LustreError: 308819:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11678.949165] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11679.190805] Lustre: fs26082-OST0001: new disk, initializing [11679.202865] Lustre: srv-fs26082-OST0001: No data found on store. Initialize space. [11679.215682] Lustre: 309370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11679.355247] Lustre: fs26082-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11686.271201] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11694.587910] LustreError: 305014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.616296] LustreError: 305014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11695.108370] Lustre: server umount fs26082-OST0000 complete [11705.443877] Lustre: server umount fs26082-OST0001 complete [11718.459879] loop1: detected capacity change from 0 to 204800000 [11724.423227] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11724.876394] Lustre: fs26082-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11731.875403] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11738.494574] loop2: detected capacity change from 0 to 204800000 [11743.762474] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11749.878367] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11758.759698] Lustre: server umount fs26082-OST0000 complete [11768.571480] Lustre: server umount fs26082-OST0001 complete [11783.148178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11783.149485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11783.155040] LustreError: Skipped 1 previous similar message [11783.179666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11783.195092] Lustre: Skipped 1 previous similar message [11788.283726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11788.291129] Lustre: Skipped 1 previous similar message [11789.494594] Lustre: server umount lustre-OST0000 complete [11795.426113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11795.447253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11795.480772] Lustre: Skipped 1 previous similar message [11795.493813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11798.523275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11798.528122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11799.951957] Lustre: server umount lustre-MDT0000 complete [11804.225402] LustreError: 302023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593819 with bad export cookie 10468617745697527658 [11804.227402] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11804.248892] LustreError: 302023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11804.762392] Lustre: server umount lustre-MDT0001 complete [11819.118644] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [11822.750704] Key type lgssc unregistered [11823.137778] LNet: 315296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11823.148878] LNetError: 315296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11823.170940] LNet: Removed LNI 192.168.202.147@tcp [11824.575141] Key type .llcrypt unregistered [11824.578494] Key type ._llcrypt unregistered [11859.589736] Key type ._llcrypt registered [11859.592158] Key type .llcrypt registered [11859.680322] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [11874.211951] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [11875.531937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11875.593397] alg: No test for adler32 (adler32-zlib) [11876.623227] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [11876.863793] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [11878.559919] Key type lgssc registered [11879.868984] Lustre: Echo OBD driver; http://www.lustre.org/ [11915.607990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11915.664087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11916.878745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11916.922140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11917.026780] Lustre: lustre-MDT0000: new disk, initializing [11917.089214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11917.125883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11922.456588] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11933.372765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11933.518394] Lustre: 319150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11933.604928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11933.614885] Lustre: Skipped 1 previous similar message [11933.783546] Lustre: lustre-MDT0001: new disk, initializing [11933.895916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11933.936554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11933.950864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11939.615404] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11950.723708] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11957.248538] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11965.272679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11965.767069] Lustre: lustre-OST0000: new disk, initializing [11965.770577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11965.790425] Lustre: 321118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11965.913497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11972.945297] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11973.209849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11973.239700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11973.442437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11984.676940] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11987.885596] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637 [11989.051943] Lustre: Unmounted lustre-client [12003.298605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12003.303398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12003.309922] Lustre: Skipped 1 previous similar message [12003.317508] Lustre: Skipped 1 previous similar message [12007.449530] Lustre: server umount lustre-OST0000 complete [12013.537440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12013.560105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12018.656768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12018.661867] Lustre: Skipped 2 previous similar messages [12022.753146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12022.760910] Lustre: Skipped 1 previous similar message [12027.359903] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12027.650972] Lustre: server umount lustre-MDT0000 complete [12032.470739] LustreError: 319141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594047 with bad export cookie 6474450476096936034 [12032.495805] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12032.889271] Lustre: server umount lustre-MDT0001 complete [12046.525652] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [12046.904917] LNet: 1 local NIs in recovery (showing 1): 192.168.202.147@tcp [12050.254901] Key type lgssc unregistered [12050.576608] LNet: 323282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12050.585338] LNetError: 323282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12117.164640] LNet: Removed LNI 192.168.202.147@tcp [12118.083968] Key type .llcrypt unregistered [12118.090253] Key type ._llcrypt unregistered [12141.134950] Key type ._llcrypt registered [12141.136849] Key type .llcrypt registered [12141.261263] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [12159.124290] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [12160.419778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12160.492800] alg: No test for adler32 (adler32-zlib) [12161.785931] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [12162.020806] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [12163.723594] Key type lgssc registered [12164.787686] Lustre: Echo OBD driver; http://www.lustre.org/ [12192.712918] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:43:26 (1788594206) [12199.725560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12199.753419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12200.991287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12201.015442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12201.118343] Lustre: lustre-MDT0000: new disk, initializing [12201.182718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12201.201462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12205.829106] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12218.422785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12218.633244] Lustre: 326496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12218.700295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12218.704716] Lustre: Skipped 1 previous similar message [12218.859801] Lustre: lustre-MDT0001: new disk, initializing [12218.993491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12219.032720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12219.040072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12224.484616] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12234.237750] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12239.945294] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12247.345811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12247.543470] Lustre: lustre-OST0000: new disk, initializing [12247.546471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12247.550581] Lustre: 328464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12247.612091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12253.248311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12253.255332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12253.373537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12254.785820] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12264.853304] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12284.385235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12284.393078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12284.402508] LustreError: Skipped 1 previous similar message [12284.403213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12284.441955] Lustre: Skipped 1 previous similar message [12287.176080] Lustre: server umount lustre-OST0000 complete [12294.628452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12294.644737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12294.647501] Lustre: Skipped 1 previous similar message [12297.700765] Lustre: server umount lustre-MDT0000 complete [12301.767984] LustreError: 326490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594317 with bad export cookie 11655062696923361773 [12301.788979] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12302.362406] Lustre: server umount lustre-MDT0001 complete [12314.507715] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [12318.253376] Key type lgssc unregistered [12318.749894] LNet: 330371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12318.770168] LNetError: 330371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12319.846543] LNet: Removed LNI 192.168.202.147@tcp [12320.807889] Key type .llcrypt unregistered [12320.810960] Key type ._llcrypt unregistered [12345.031775] Key type ._llcrypt registered [12345.041286] Key type .llcrypt registered [12345.249524] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [12346.676723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12346.794090] alg: No test for adler32 (adler32-zlib) [12348.131558] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [12348.491468] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [12350.199168] Key type lgssc registered [12352.106739] Lustre: Echo OBD driver; http://www.lustre.org/ [12362.907714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12362.969731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12364.729574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12370.426884] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12373.993558] LustreError: 331639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12374.015392] LustreError: 331639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12379.106115] LustreError: 331640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12380.297655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12380.778167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12386.184194] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12394.565894] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12400.495831] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12407.395470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12407.726307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12414.148605] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12422.022780] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12436.120370] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12437.826816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12442.081668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12442.095780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12442.117544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12445.245636] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.47@tcp (stopping) [12445.666869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12447.981902] Lustre: server umount lustre-MDT0000 complete [12450.304207] LustreError: 331634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12452.405181] LustreError: 331619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594467 with bad export cookie 17903776582904792834 [12452.421100] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12452.433054] LustreError: 331619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12455.420156] LustreError: 331634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12455.444836] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.47@tcp (stopping) [12455.454564] LustreError: 331634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12455.489829] Lustre: Skipped 3 previous similar messages [12455.911602] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12455.929592] Lustre: Skipped 1 previous similar message [12457.952577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12457.955093] Lustre: Skipped 1 previous similar message [12459.037644] Lustre: server umount lustre-MDT0001 complete [12463.257591] Lustre: server umount lustre-OST0000 complete [12472.063803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12476.856944] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12479.781451] Lustre: 335820:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.147@tcp [12482.340550] Lustre: server umount MGS complete [12491.094073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12491.718763] LustreError: 336394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12491.744736] LustreError: 336394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12491.862581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12496.745633] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12500.285266] LustreError: 336391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.321417] LustreError: 336391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12506.594379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12507.275815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12513.384454] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12523.233162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12523.560063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12529.892664] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12537.189993] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12539.167812] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12545.738162] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12547.452317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12552.678752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12552.697466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12552.701201] Lustre: Skipped 1 previous similar message [12558.382438] Lustre: server umount lustre-OST0000 complete [12562.912865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12562.949485] Lustre: Skipped 1 previous similar message [12562.957613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12562.966130] Lustre: Skipped 3 previous similar messages [12568.699688] Lustre: server umount lustre-MDT0000 complete [12572.967953] LustreError: 336376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594588 with bad export cookie 17903776582904793674 [12572.974748] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12572.981853] LustreError: 336376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12573.516647] Lustre: server umount lustre-MDT0001 complete [12586.392440] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [12589.932366] Key type lgssc unregistered [12590.314221] LNet: 339832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12590.333525] LNetError: 339832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12590.371276] LNet: Removed LNI 192.168.202.147@tcp [12591.403959] Key type .llcrypt unregistered [12591.406312] Key type ._llcrypt unregistered [12609.664511] Key type ._llcrypt registered [12609.667154] Key type .llcrypt registered [12609.810129] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [12626.126324] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [12627.442485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12627.684032] alg: No test for adler32 (adler32-zlib) [12628.686713] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [12628.977743] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [12630.679205] Key type lgssc registered [12632.101511] Lustre: Echo OBD driver; http://www.lustre.org/ [12669.038711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12669.081303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12670.357130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12670.395388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12670.497732] Lustre: lustre-MDT0000: new disk, initializing [12670.661604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12670.717646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12675.387188] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12686.094662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12686.247463] Lustre: 342993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12686.305711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12686.308511] Lustre: Skipped 1 previous similar message [12686.404165] Lustre: lustre-MDT0001: new disk, initializing [12686.468607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12686.495968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12686.508246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12691.882571] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12702.591191] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12710.207429] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12719.278273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12719.800909] Lustre: lustre-OST0000: new disk, initializing [12719.815727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12719.837665] Lustre: 344958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12720.042981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12726.452238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12726.471184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12726.706957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12730.423464] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12740.482695] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12744.974079] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12745.174177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12749.506230] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12749.784663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12752.359061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12752.363636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12752.365438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12752.381771] Lustre: Skipped 1 previous similar message [12752.394423] LustreError: Skipped 1 previous similar message [12757.478029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12757.496828] Lustre: Skipped 2 previous similar messages [12762.595170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12762.612390] Lustre: Skipped 1 previous similar message [12766.175489] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12766.391358] Lustre: server umount lustre-OST0000 complete [12772.836861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12772.844647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12772.850515] Lustre: Skipped 1 previous similar message [12772.870790] Lustre: Skipped 1 previous similar message [12777.957339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12777.973810] Lustre: Skipped 1 previous similar message [12785.119319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12785.348875] Lustre: server umount lustre-MDT0000 complete [12789.102266] LustreError: 344957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594804 with bad export cookie 6421121633756937580 [12789.107410] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12789.117814] LustreError: 344957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12789.409463] Lustre: server umount lustre-MDT0001 complete [12798.306290] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:53:32 (1788594812) [12815.374645] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [12825.042662] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [12862.448867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12862.745937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12862.767027] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12862.908418] Lustre: lustre-MDT0000: new disk, initializing [12863.096394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12863.132077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12869.316364] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12880.452752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12880.601380] Lustre: 349721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12880.674330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12880.677146] Lustre: Skipped 1 previous similar message [12880.770886] Lustre: lustre-MDT0001: new disk, initializing [12880.916712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12880.967526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12880.984449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12885.725380] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12894.880647] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12900.126601] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12907.078356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12907.434401] Lustre: lustre-OST0000: new disk, initializing [12907.440927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12907.446730] Lustre: 351688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12907.593407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12914.258827] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12915.256334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12915.273164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12915.381072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12923.510611] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12955.833364] Lustre: server umount lustre-OST0000 complete [12961.761027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12961.766258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12961.777852] Lustre: Skipped 1 previous similar message [12961.797468] Lustre: Skipped 3 previous similar messages [12970.099226] Lustre: server umount lustre-MDT0000 complete [12971.947976] Lustre: 341213:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788594971/real 1788594971] req@0000000027ba227b x1875477417521408/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788594987 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12971.982538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12972.004295] LustreError: 349731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12972.031154] LustreError: 349731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12974.231314] LustreError: 349711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594989 with bad export cookie 6421121633756938462 [12974.237674] LustreError: 349711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12974.238116] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12974.747302] Lustre: server umount lustre-MDT0001 complete [12986.438542] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [12988.948264] Key type lgssc unregistered [12989.241142] LNet: 353842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12989.256299] LNetError: 353842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12989.272887] LNet: Removed LNI 192.168.202.147@tcp [12990.123883] Key type .llcrypt unregistered [12990.127774] Key type ._llcrypt unregistered [13089.753554] Key type ._llcrypt registered [13089.754684] Key type .llcrypt registered [13089.863103] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13090.163514] Key type .llcrypt unregistered [13090.165905] Key type ._llcrypt unregistered [13174.371935] Key type ._llcrypt registered [13174.374508] Key type .llcrypt registered [13174.551025] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13174.967553] Key type .llcrypt unregistered [13174.968827] Key type ._llcrypt unregistered [13205.098894] Key type ._llcrypt registered [13205.101244] Key type .llcrypt registered [13205.208314] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13205.504451] Key type .llcrypt unregistered [13205.513803] Key type ._llcrypt unregistered [13232.258961] Key type ._llcrypt registered [13232.261067] Key type .llcrypt registered [13232.377398] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13232.761551] Key type .llcrypt unregistered [13232.763606] Key type ._llcrypt unregistered [13317.551733] Key type ._llcrypt registered [13317.556843] Key type .llcrypt registered [13317.681581] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13318.081392] Key type .llcrypt unregistered [13318.086746] Key type ._llcrypt unregistered [13353.794457] Key type ._llcrypt registered [13353.796706] Key type .llcrypt registered [13353.937207] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13354.388661] Key type .llcrypt unregistered [13354.395069] Key type ._llcrypt unregistered [13388.193203] Key type ._llcrypt registered [13388.194760] Key type .llcrypt registered [13388.304589] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13388.635392] Key type .llcrypt unregistered [13388.637090] Key type ._llcrypt unregistered [13422.939194] Key type ._llcrypt registered [13422.941750] Key type .llcrypt registered [13423.081133] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13423.427419] Key type .llcrypt unregistered [13423.429737] Key type ._llcrypt unregistered [13457.272965] Key type ._llcrypt registered [13457.279385] Key type .llcrypt registered [13457.373143] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13457.578368] Key type .llcrypt unregistered [13457.582728] Key type ._llcrypt unregistered [13484.963934] Key type ._llcrypt registered [13484.965913] Key type .llcrypt registered [13485.099329] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13485.433325] Key type .llcrypt unregistered [13485.437862] Key type ._llcrypt unregistered [13546.224471] Key type ._llcrypt registered [13546.229949] Key type .llcrypt registered [13546.359915] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13546.711881] Key type .llcrypt unregistered [13546.713826] Key type ._llcrypt unregistered [13577.614779] Key type ._llcrypt registered [13577.616946] Key type .llcrypt registered [13577.764720] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13578.134333] Key type .llcrypt unregistered [13578.137199] Key type ._llcrypt unregistered [13608.381106] Key type ._llcrypt registered [13608.384813] Key type .llcrypt registered [13608.526866] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [13608.804992] Key type .llcrypt unregistered [13608.807378] Key type ._llcrypt unregistered [13664.358977] Key type ._llcrypt registered [13664.361781] Key type .llcrypt registered [13664.456281] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [13681.177232] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [13682.486310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13682.565993] alg: No test for adler32 (adler32-zlib) [13683.850781] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [13684.111228] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [13685.762089] Key type lgssc registered [13687.120738] Lustre: Echo OBD driver; http://www.lustre.org/ [13725.818847] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13725.848678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13727.141836] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13727.173461] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13727.262795] Lustre: tst250fs-MDT0000: new disk, initializing [13727.371314] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13727.397971] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13731.884239] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [13743.482959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13743.698242] Lustre: 374102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13743.803299] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13743.817181] Lustre: Skipped 1 previous similar message [13743.927505] Lustre: tst250fs-MDT0001: new disk, initializing [13744.086116] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13744.142431] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13744.165411] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13749.456189] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [13755.002388] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13764.630712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13765.063286] Lustre: tst250fs-OST0000: new disk, initializing [13765.073373] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13765.082149] Lustre: 375733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13765.246944] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13771.389373] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13771.415764] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13771.573367] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13773.052732] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [13784.918491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13785.098533] Lustre: tst250fs-OST0001: new disk, initializing [13785.103426] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13785.110457] Lustre: 376605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13785.258078] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13792.274648] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [13794.447615] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13794.462226] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13794.588189] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13798.853687] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13800.880615] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13800.885337] Lustre: Skipped 1 previous similar message [13820.900656] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13820.914107] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13820.931362] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13825.508831] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13825.515745] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13825.526045] Lustre: Skipped 1 previous similar message [13825.541687] Lustre: Skipped 3 previous similar messages [13829.058818] Lustre: server umount tst250fs-MDT0000 complete [13832.755704] LustreError: 375362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595848 with bad export cookie 14400913242219717140 [13832.767195] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13832.777241] LustreError: 375362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13833.170347] Lustre: server umount tst250fs-MDT0001 complete [13846.045641] Lustre: server umount tst250fs-OST0000 complete [13860.755785] Lustre: server umount tst250fs-OST0001 complete [13883.776374] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [13894.228663] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [13925.345071] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:12:19 (1788595939) [13927.035482] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13945.699514] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [13956.806786] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [13990.515665] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13684 sec ========= 04:13:24 (1788596004) [13991.973740] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:13:26 (1788596006) === [13995.276536] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:13:29 (1788596009) === [14040.798824] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [14043.914818] Key type lgssc unregistered [14044.321950] LNet: 385749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14044.329881] LNetError: 385749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14044.366679] LNet: Removed LNI 192.168.202.147@tcp [14045.279182] Key type .llcrypt unregistered [14045.284385] Key type ._llcrypt unregistered