[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511007640 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002007] Switched APIC routing to physical x2apic. [ 0.003011] kvm-guest: setup PV IPIs [ 0.006348] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010104] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.021657] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025843] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028100] *** VALIDATE tmpfs *** [ 0.029463] *** VALIDATE proc *** [ 0.030162] *** VALIDATE cgroup *** [ 0.031004] *** VALIDATE cgroup2 *** [ 0.032260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035020] Spectre V2 : User space: Vulnerable [ 0.036003] Speculative Store Bypass: Vulnerable [ 0.039304] debug: unmapping init [mem 0xffffffff95459000-0xffffffff95460fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042550] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043019] ... version: 2 [ 0.044010] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.046007] ... value mask: 0000ffffffffffff [ 0.047008] ... max period: 00007fffffffffff [ 0.048011] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050294] rcu: Hierarchical SRCU implementation. [ 0.052561] smp: Bringing up secondary CPUs ... [ 0.053527] x86: Booting SMP configuration: [ 0.054019] .... node #0, CPUs: #1 #2 #3 [ 0.060053] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063009] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.130706] node 0 deferred pages initialised in 66ms [ 0.134178] devtmpfs: initialized [ 0.136282] x86/mm: Memory block size: 128MB [ 0.139843] gcov: version magic: 0x41383552 [ 0.142254] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149281] pinctrl core: initialized pinctrl subsystem [ 0.152113] [ 0.152871] ************************************************************* [ 0.155010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157006] ** ** [ 0.159009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161008] ** ** [ 0.164007] ** This means that this kernel is built to expose internal ** [ 0.165000] ** IOMMU data structures, which may compromise security on ** [ 0.166008] ** your system. ** [ 0.168008] ** ** [ 0.171007] ** If you see this message and you are not debugging the ** [ 0.173008] ** kernel, report this immediately to your vendor! ** [ 0.175008] ** ** [ 0.176010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178010] ************************************************************* [ 0.180971] NET: Registered protocol family 16 [ 0.182398] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.184041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.190085] cpuidle: using governor menu [ 0.192000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193515] PCI: Using configuration type 1 for base access [ 0.195140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202083] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206068] cryptd: max_cpu_qlen set to 1000 [ 0.208143] ACPI: Added _OSI(Module Device) [ 0.209013] ACPI: Added _OSI(Processor Device) [ 0.211016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218464] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224450] ACPI: Interpreter enabled [ 0.226194] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228009] ACPI: Using IOAPIC for interrupt routing [ 0.230099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234460] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246111] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.249045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.251015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256149] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.262687] acpiphp: Slot [2] registered [ 0.264239] acpiphp: Slot [5] registered [ 0.266121] acpiphp: Slot [6] registered [ 0.267098] acpiphp: Slot [7] registered [ 0.269094] acpiphp: Slot [8] registered [ 0.270096] acpiphp: Slot [9] registered [ 0.272117] acpiphp: Slot [10] registered [ 0.273080] acpiphp: Slot [3] registered [ 0.275073] acpiphp: Slot [4] registered [ 0.276067] acpiphp: Slot [11] registered [ 0.278069] acpiphp: Slot [12] registered [ 0.279115] acpiphp: Slot [13] registered [ 0.281122] acpiphp: Slot [14] registered [ 0.282064] acpiphp: Slot [15] registered [ 0.284114] acpiphp: Slot [16] registered [ 0.285070] acpiphp: Slot [17] registered [ 0.287073] acpiphp: Slot [18] registered [ 0.289071] acpiphp: Slot [19] registered [ 0.290094] acpiphp: Slot [20] registered [ 0.292072] acpiphp: Slot [21] registered [ 0.294198] acpiphp: Slot [22] registered [ 0.296062] acpiphp: Slot [23] registered [ 0.297062] acpiphp: Slot [24] registered [ 0.298060] acpiphp: Slot [25] registered [ 0.300085] acpiphp: Slot [26] registered [ 0.301063] acpiphp: Slot [27] registered [ 0.302063] acpiphp: Slot [28] registered [ 0.304077] acpiphp: Slot [29] registered [ 0.305058] acpiphp: Slot [30] registered [ 0.306127] acpiphp: Slot [31] registered [ 0.308048] PCI host bridge to bus 0000:00 [ 0.310025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.316031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.327188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334286] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.343016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.347613] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355028] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.357707] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362833] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.379013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.385021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390281] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.407014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436355] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.441010] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.451017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.478545] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.484012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.489011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.500012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.508947] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.513010] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.517011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.534016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.551322] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.556941] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.566015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.581014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591507] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.599014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.605015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.626013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.636766] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.639914] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.641195] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.642248] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.644126] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.649025] iommu: Default domain type: Passthrough [ 0.650328] SCSI subsystem initialized [ 0.652129] ACPI: bus type USB registered [ 0.653071] usbcore: registered new interface driver usbfs [ 0.655051] usbcore: registered new interface driver hub [ 0.657045] usbcore: registered new device driver usb [ 0.659157] pps_core: LinuxPPS API ver. 1 registered [ 0.661010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.664049] PTP clock support registered [ 0.666135] EDAC MC: Ver: 3.0.0 [ 0.667496] PCI: Using ACPI for IRQ routing [ 0.668511] NetLabel: Initializing [ 0.669000] NetLabel: domain hash size = 128 [ 0.671016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.672051] NetLabel: unlabeled traffic allowed by default [ 0.674081] vgaarb: loaded [ 0.675227] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.678014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.683385] clocksource: Switched to clocksource kvm-clock [ 0.791980] VFS: Disk quotas dquot_6.6.0 [ 0.793449] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.795565] *** VALIDATE ramfs *** [ 0.796772] *** VALIDATE hugetlbfs *** [ 0.798585] pnp: PnP ACPI init [ 0.801785] pnp: PnP ACPI: found 6 devices [ 0.822163] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.825521] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.827154] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.828874] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.831395] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.833887] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.836853] NET: Registered protocol family 2 [ 0.839768] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.844740] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.848474] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.853970] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.857223] TCP: Hash tables configured (established 65536 bind 65536) [ 0.859809] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.862597] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.865123] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.867915] NET: Registered protocol family 1 [ 0.870390] RPC: Registered named UNIX socket transport module. [ 0.872666] RPC: Registered udp transport module. [ 0.874349] RPC: Registered tcp transport module. [ 0.876534] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.878813] NET: Registered protocol family 44 [ 0.880616] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.882956] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.885113] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.887463] PCI: CLS 0 bytes, default 64 [ 0.889085] Unpacking initramfs... [ 2.215703] debug: unmapping init [mem 0xffff99813cc54000-0xffff99813ffbffff] [ 2.220269] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.222778] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.226090] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.683041] Initialise system trusted keyrings [ 2.684655] Key type blacklist registered [ 2.686917] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.695264] zbud: loaded [ 2.698496] *** VALIDATE nfs *** [ 2.699827] *** VALIDATE nfs4 *** [ 2.701480] pstore: using deflate compression [ 2.705363] Platform Keyring initialized [ 2.783894] NET: Registered protocol family 38 [ 2.785399] Key type asymmetric registered [ 2.786596] Asymmetric key parser 'x509' registered [ 2.787975] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.790881] io scheduler mq-deadline registered [ 2.792258] io scheduler kyber registered [ 2.793809] io scheduler bfq registered [ 2.795449] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.797607] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.800531] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.802834] ACPI: Power Button [PWRF] [ 2.807982] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.814586] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.826646] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.831843] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.844807] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.871626] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.899077] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.903292] Non-volatile memory driver v1.3 [ 2.905032] Linux agpgart interface v0.103 [ 2.929183] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.931887] vda: detected capacity change from 0 to 74706944 [ 2.944240] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.946676] vdb: detected capacity change from 0 to 1073741824 [ 2.959114] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.961369] vdc: detected capacity change from 0 to 2621440000 [ 2.972086] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.974380] vdd: detected capacity change from 0 to 2621440000 [ 2.986744] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.989163] vde: detected capacity change from 0 to 4294967296 [ 3.003124] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.006294] vdf: detected capacity change from 0 to 4294967296 [ 3.014299] libphy: Fixed MDIO Bus: probed [ 3.018939] usbcore: registered new interface driver usbserial_generic [ 3.021107] usbserial: USB Serial support registered for generic [ 3.023492] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.027515] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.029154] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.031772] mousedev: PS/2 mouse device common for all mice [ 3.034274] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.038772] rtc_cmos 00:05: RTC can wake from S4 [ 3.044608] rtc_cmos 00:05: registered as rtc0 [ 3.044979] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.046732] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.053018] intel_pstate: CPU model not supported [ 3.054953] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.057930] hid: raw HID events driver (C) Jiri Kosina [ 3.061162] usbcore: registered new interface driver usbhid [ 3.063304] usbhid: USB HID core driver [ 3.064981] drop_monitor: Initializing network drop monitor service [ 3.067290] Initializing XFRM netlink socket [ 3.069229] NET: Registered protocol family 10 [ 3.076567] Segment Routing with IPv6 [ 3.078874] NET: Registered protocol family 17 [ 3.080922] mpls_gso: MPLS GSO support [ 3.086960] RAS: Correctable Errors collector initialized. [ 3.090039] AVX version of gcm_enc/dec engaged. [ 3.091541] AES CTR mode by8 optimization enabled [ 3.152915] sched_clock: Marking stable (3152894897, 0)->(3980144546, -827249649) [ 3.155907] registered taskstats version 1 [ 3.157595] Loading compiled-in X.509 certificates [ 3.159241] zswap: loaded using pool lzo/zbud [ 3.187439] Key type big_key registered [ 3.200211] Key type encrypted registered [ 3.202027] ima: No TPM chip found, activating TPM-bypass! [ 3.204471] ima: Allocated hash algorithm: sha1 [ 3.206507] ima: No architecture policies found [ 3.208402] evm: Initialising EVM extended attributes: [ 3.210213] evm: security.selinux [ 3.211104] evm: security.ima [ 3.211944] evm: security.capability [ 3.212845] evm: HMAC attrs: 0x1 [ 3.214742] rtc_cmos 00:05: setting system clock to 2026-08-17 04:05:43 UTC (1786939543) [ 3.219679] debug: unmapping init [mem 0xffffffff96403000-0xffffffff965fffff] [ 3.222347] debug: unmapping init [mem 0xffffffff95182000-0xffffffff95458fff] [ 3.231122] Write protecting the kernel read-only data: 28672k [ 3.235177] debug: unmapping init [mem 0xffffffff93803000-0xffffffff939fffff] [ 3.237781] debug: unmapping init [mem 0xffffffff94114000-0xffffffff941fffff] [ 3.273056] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.281038] systemd[1]: Detected virtualization kvm. [ 3.282775] systemd[1]: Detected architecture x86-64. [ 3.284521] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.310570] systemd[1]: No hostname configured. [ 3.312216] systemd[1]: Set hostname to . [ 3.314239] random: systemd: uninitialized urandom read (16 bytes read) [ 3.316835] systemd[1]: Initializing machine ID from random generator. [ 3.358560] random: ln: uninitialized urandom read (6 bytes read) [ 3.434316] random: systemd: uninitialized urandom read (16 bytes read) [ 3.436844] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.445447] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.450295] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.006344] device-mapper: uevent: version 1.0.3 [ 4.008822] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.673600] random: fast init done [ 4.693043] virtio_net virtio0 ens2: renamed from eth0 [ 4.734360] scsi host0: ata_piix [ 4.742213] scsi host1: ata_piix [ 4.753632] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.756605] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.969946] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.718305] random: crng init done [ 9.719658] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.944056] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.940935] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.124021] SELinux: Disabled at runtime. [ 11.176759] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.183204] systemd[1]: Detected virtualization kvm. [ 11.184433] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.637427] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.641208] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.646273] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.649585] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.653190] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.662211] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.671933] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ 11.739049] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.056417] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.272570] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.328190] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.430237] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.449226] EDAC sbridge: Ver: 1.1.2 [ 13.941254] Key type dns_resolver registered [ 14.232399] NFS: Registering the id_resolver key type [ 14.234223] Key type id_resolver registered [ 14.235950] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg424-server login: [ 36.306274] libcfs: loading out-of-tree module taints kernel. [ 36.322836] Key type ._llcrypt registered [ 36.323894] Key type .llcrypt registered [ 36.360955] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 43.983899] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 44.595144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 44.600833] alg: No test for adler32 (adler32-zlib) [ 45.602117] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 45.912644] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 47.520155] Key type lgssc registered [ 48.096704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 54.214069] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 67.509612] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 73.004376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 73.018594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 74.162317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 74.186231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 74.229336] Lustre: lustre-MDT0000: new disk, initializing [ 74.270840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 74.281298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 77.519860] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 85.829193] hrtimer: interrupt took 3031874 ns [ 87.716114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.810707] Lustre: 6479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 87.833141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 87.836791] Lustre: Skipped 1 previous similar message [ 87.901728] Lustre: lustre-MDT0001: new disk, initializing [ 87.964626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 87.988957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 88.005289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 91.456331] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.535675] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 102.955277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 103.234816] Lustre: lustre-OST0000: new disk, initializing [ 103.239187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 103.248804] Lustre: 8417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 103.321171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 108.823340] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.599925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 109.611489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 109.657192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 116.759770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 116.829309] Lustre: lustre-OST0001: new disk, initializing [ 116.832466] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 116.836187] Lustre: 9487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 116.884260] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 119.351196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 119.355170] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 119.376072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 119.913308] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.208090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 130.563184] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 133.583862] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing check_logdir /tmp/testlogs/ [ 135.750649] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing yml_node [ 137.711164] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 138.879654] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 139.974521] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 140.638162] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 00:07:59 EDT 2026 [ 148.026950] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 148.722451] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 164.832772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 164.836343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 164.841795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 165.345736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.345886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 167.177264] Lustre: server umount lustre-MDT0000 complete [ 169.168667] LustreError: 10363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939709 with bad export cookie 11202524095965735213 [ 169.177112] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 169.188173] LustreError: 10363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 170.464978] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 170.468257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 170.475805] Lustre: Skipped 3 previous similar messages [ 170.486068] Lustre: Skipped 4 previous similar messages [ 174.498724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 174.501774] Lustre: Skipped 1 previous similar message [ 175.370926] Lustre: server umount lustre-MDT0001 complete [ 183.307203] Lustre: server umount lustre-OST0000 complete [ 191.390480] Lustre: server umount lustre-OST0001 complete [ 194.575781] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 199.282491] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 216.924584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.055890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.084626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.132230] Lustre: lustre-MDT0000: new disk, initializing [ 217.174978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.187801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.339423] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 224.679152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.750024] Lustre: 15031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 224.887656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.871900] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 232.202237] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 235.079380] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 238.743657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.915545] Lustre: lustre-OST0000: new disk, initializing [ 238.920262] Lustre: Skipped 1 previous similar message [ 238.925988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.930948] Lustre: Skipped 2 previous similar messages [ 238.935890] Lustre: 16999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.986040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.989429] Lustre: Skipped 1 previous similar message [ 240.552814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.556874] Lustre: Skipped 1 previous similar message [ 240.559886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.596705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.262771] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 247.722375] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 249.766631] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 249.870510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 252.145695] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 252.277426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 255.972081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 255.973889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.980315] LustreError: Skipped 1 previous similar message [ 255.990934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 255.994355] Lustre: Skipped 1 previous similar message [ 259.484533] Lustre: server umount lustre-OST0000 complete [ 265.697491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 265.701388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.708622] Lustre: Skipped 1 previous similar message [ 265.710831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.317989] Lustre: server umount lustre-MDT0000 complete [ 269.027340] LustreError: 15023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939809 with bad export cookie 11202524095965736746 [ 269.031269] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.033829] LustreError: 15023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 269.167033] Lustre: server umount lustre-MDT0001 complete [ 271.423746] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 00:10:10 (1786939810) [ 274.471421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.688525] LustreError: 19259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.700191] LustreError: 19259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 274.743857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 276.615968] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 280.033622] LustreError: 19260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.582593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 282.526868] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 285.760930] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 288.117452] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 290.894996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 290.981233] LustreError: 21075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.369796] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 296.386882] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 303.555844] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 309.732636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.735037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.739991] Lustre: Skipped 3 previous similar messages [ 309.742597] Lustre: Skipped 3 previous similar messages [ 313.117374] Lustre: server umount lustre-MDT0000 complete [ 314.782584] LustreError: 21099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939855 with bad export cookie 11202524095965737628 [ 314.786406] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.788076] LustreError: 21099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 314.849287] LustreError: 19255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.940063] Lustre: server umount lustre-MDT0001 complete [ 322.783310] Lustre: server umount lustre-OST0000 complete [ 331.795847] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 336.052134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 336.201469] LustreError: 23633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.220073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 336.223644] Lustre: Skipped 2 previous similar messages [ 337.797782] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 341.201109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.033760] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 344.279446] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 347.053848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 349.572447] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 353.056229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 353.136982] Lustre: lustre-OST0001: new disk, initializing [ 353.139203] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 353.142967] Lustre: 25949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 354.767972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 354.772650] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 354.822060] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 355.495447] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 360.216276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 361.632180] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 366.049935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.053408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.056589] Lustre: Skipped 4 previous similar messages [ 366.065337] Lustre: Skipped 5 previous similar messages [ 371.052321] Lustre: server umount lustre-MDT0000 complete [ 371.169149] LustreError: 24363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.177401] LustreError: 24363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 372.567490] LustreError: 23616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939912 with bad export cookie 11202524095965738475 [ 372.571023] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.572741] LustreError: 23616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 372.744687] Lustre: server umount lustre-MDT0001 complete [ 384.523429] Lustre: server umount lustre-OST0000 complete [ 396.391589] Lustre: server umount lustre-OST0001 complete [ 402.461951] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 406.236588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.382947] LustreError: 28906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.402640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.404457] Lustre: Skipped 3 previous similar messages [ 407.932858] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 411.163354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.728479] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 413.728675] Lustre: 30054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 416.059441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.290837] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 421.288593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.463464] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 426.724041] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 427.988060] Lustre: 31931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 431.585889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.590473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 431.591852] Lustre: Skipped 2 previous similar messages [ 431.593849] Lustre: Skipped 1 previous similar message [ 435.867541] Lustre: server umount lustre-MDT0000 complete [ 437.117245] LustreError: 29658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939977 with bad export cookie 11202524095965739707 [ 437.120065] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 437.121836] LustreError: 29658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 437.231169] Lustre: server umount lustre-MDT0001 complete [ 448.003586] Lustre: server umount lustre-OST0000 complete [ 459.616454] Lustre: server umount lustre-OST0001 complete [ 462.764128] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 00:13:21 (1786940001) [ 472.102470] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 475.603666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.753705] LustreError: 34727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.763583] LustreError: 34727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 477.043347] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 479.719778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.220139] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 482.232350] Lustre: 35875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 484.474175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 486.491868] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 489.391526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.530097] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 494.760116] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 496.128627] Lustre: 37752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 500.193024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 500.196617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.202457] Lustre: Skipped 1 previous similar message [ 503.644307] Lustre: server umount lustre-MDT0000 complete [ 504.820209] LustreError: 37754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940045 with bad export cookie 11202524095965740792 [ 504.823192] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.824811] LustreError: 37754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 504.933396] Lustre: server umount lustre-MDT0001 complete [ 516.358675] Lustre: server umount lustre-OST0000 complete [ 526.874268] Lustre: server umount lustre-OST0001 complete [ 529.706771] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 00:14:28 (1786940068) [ 534.919731] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 538.450898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.601356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.604402] Lustre: Skipped 7 previous similar messages [ 539.943388] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 542.701492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.106518] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 545.053279] Lustre: 41034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 547.116112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 548.324451] LustreError: 41581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.328592] LustreError: 41581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 549.158678] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 552.014784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 554.172743] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 557.443215] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 559.257654] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 559.260545] Lustre: Skipped 1 previous similar message [ 563.169244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.173343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.177054] Lustre: Skipped 5 previous similar messages [ 566.651967] Lustre: server umount lustre-MDT0000 complete [ 568.208698] LustreError: 40637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940108 with bad export cookie 11202524095965741877 [ 568.211678] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.213617] LustreError: 40637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 574.388589] Lustre: server umount lustre-MDT0001 complete [ 577.245570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.016733] LustreError: 3613:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9981bc3d5500 x1873741963625728/t0(0) o250->MGC192.168.204.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 579.491970] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 582.373917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.965147] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 586.365812] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 597.464491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [ 597.927107] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 599.555398] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 600.132925] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 603.109518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 607.642418] Lustre: server umount lustre-MDT0000 complete [ 608.922334] LustreError: 40637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940149 with bad export cookie 11202524095965742955 [ 608.927821] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.066609] Lustre: server umount lustre-MDT0001 complete [ 620.546941] Lustre: server umount lustre-OST0000 complete [ 632.169826] Lustre: server umount lustre-OST0001 complete [ 635.040884] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 00:16:14 (1786940174) [ 639.922608] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 643.414885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.961222] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 647.921320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.627107] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 650.637910] Lustre: 48677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 650.641181] Lustre: 48677:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 652.892554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 655.066915] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 657.786415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 659.764906] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 662.788500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 664.273781] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 668.640713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 668.642732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 668.646366] Lustre: Skipped 11 previous similar messages [ 674.715085] Lustre: server umount lustre-MDT0000 complete [ 675.849454] LustreError: 50669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940216 with bad export cookie 11202524095965743753 [ 675.855315] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.855635] LustreError: 50669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 675.967996] Lustre: server umount lustre-MDT0001 complete [ 687.616722] Lustre: server umount lustre-OST0000 complete [ 699.162480] Lustre: server umount lustre-OST0001 complete [ 702.020944] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 00:17:21 (1786940241) [ 702.786146] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 703.315539] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 00:17:22 (1786940242) [ 706.366475] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 708.854850] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 715.729263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.813372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 715.854179] Lustre: lustre-MDT0000: new disk, initializing [ 715.877188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 717.088863] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 720.497440] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 722.577750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 722.684403] Lustre: 54743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 724.488820] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 724.603225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 724.610268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 727.979208] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 728.608510] Lustre: DEBUG MARKER: create test files [ 739.808849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 743.874690] Lustre: server umount lustre-OST0000 complete [ 745.170549] Lustre: server umount lustre-MDT0000 complete [ 748.435959] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 749.386204] Key type lgssc unregistered [ 749.515553] LNet: 56351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.517678] LNetError: 56351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.524272] LNet: Removed LNI 192.168.204.124@tcp [ 749.837090] Key type .llcrypt unregistered [ 749.837997] Key type ._llcrypt unregistered [ 759.750115] Key type ._llcrypt registered [ 759.751239] Key type .llcrypt registered [ 759.789914] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 760.071773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.080108] alg: No test for adler32 (adler32-zlib) [ 760.921822] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 761.002613] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 762.576126] Key type lgssc registered [ 762.903514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 765.267814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 765.274481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.402913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 767.620230] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 769.820252] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 771.879591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 771.980220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.755959] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 775.977379] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 776.100811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 776.122321] Lustre: 59059:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 776.125361] Lustre: 59059:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 776.128058] Lustre: 59059:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 776.130603] Lustre: 59059:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 776.133125] Lustre: 59059:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 776.135399] Lustre: 59059:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 778.887239] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 780.341892] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 786.402779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 786.402813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.412341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 788.177716] Lustre: server umount lustre-OST0000 complete [ 789.400071] Lustre: server umount lustre-MDT0000 complete [ 792.611413] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 793.529899] Key type lgssc unregistered [ 793.642365] LNet: 60668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.644427] LNetError: 60668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.651279] LNet: Removed LNI 192.168.204.124@tcp [ 793.926565] Key type .llcrypt unregistered [ 793.927891] Key type ._llcrypt unregistered [ 803.921193] Key type ._llcrypt registered [ 803.922634] Key type .llcrypt registered [ 803.962630] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 804.474630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.490935] alg: No test for adler32 (adler32-zlib) [ 805.323518] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 805.399730] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 806.968130] Key type lgssc registered [ 807.269385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 809.539990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 809.545897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.670955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.833271] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 813.924676] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 815.982430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 816.078229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 817.872612] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 820.071726] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 820.202404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 820.224952] Lustre: 63474:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 820.229675] Lustre: 63474:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 820.233207] Lustre: 63474:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 820.236162] Lustre: 63474:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 820.239404] Lustre: 63474:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 820.242296] Lustre: 63474:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 823.014573] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 825.313491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 825.313519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.319891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 826.954269] Lustre: server umount lustre-OST0000 complete [ 830.252495] Lustre: server umount lustre-MDT0000 complete [ 833.633188] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 834.575683] Key type lgssc unregistered [ 834.694310] LNet: 65032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.696520] LNetError: 65032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.708282] LNet: Removed LNI 192.168.204.124@tcp [ 835.006535] Key type .llcrypt unregistered [ 835.007828] Key type ._llcrypt unregistered [ 840.516787] Key type ._llcrypt registered [ 840.517667] Key type .llcrypt registered [ 840.553861] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 846.465113] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 846.747259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 846.758559] alg: No test for adler32 (adler32-zlib) [ 847.605937] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 847.689899] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 849.264155] Key type lgssc registered [ 849.623893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 859.996812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.003085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.079299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 861.088163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 861.120989] Lustre: lustre-MDT0000: new disk, initializing [ 861.142128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.147600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 862.382688] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 866.564920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.616503] Lustre: 68155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 866.640578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 866.643376] Lustre: Skipped 1 previous similar message [ 866.683276] Lustre: lustre-MDT0001: new disk, initializing [ 866.719538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 866.732693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 866.737693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 868.125598] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 871.652890] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 873.329105] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 875.390399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.490633] Lustre: lustre-OST0000: new disk, initializing [ 875.492490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 875.494741] Lustre: 70122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 875.523052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.230186] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 880.745573] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 882.032027] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 884.725694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 884.729399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 884.754611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 885.146311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 886.351920] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 886.414346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 889.826909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.827055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.827616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.832536] Lustre: Skipped 1 previous similar message [ 889.835850] LustreError: Skipped 1 previous similar message [ 893.142497] Lustre: server umount lustre-OST0000 complete [ 894.945862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.946797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 894.950439] Lustre: Skipped 1 previous similar message [ 894.952789] Lustre: Skipped 1 previous similar message [ 900.065138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 900.067039] Lustre: Skipped 2 previous similar messages [ 900.567159] Lustre: server umount lustre-MDT0000 complete [ 901.713161] LustreError: 68193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940441 with bad export cookie 6293341864371717064 [ 901.715833] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.716768] LustreError: 68193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 901.822885] Lustre: server umount lustre-MDT0001 complete [ 904.387695] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 00:20:43 (1786940443) [ 907.602254] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 919.317313] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 922.136635] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 932.853426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.927233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 932.937920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 932.972563] Lustre: lustre-MDT0000: new disk, initializing [ 932.994650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.001396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 934.357340] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 938.311604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.352209] Lustre: 76151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 938.376560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 938.379457] Lustre: Skipped 1 previous similar message [ 938.416403] Lustre: lustre-MDT0001: new disk, initializing [ 938.453834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.467896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 938.471799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 939.820502] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 943.510697] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 945.258487] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 947.402541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.506660] Lustre: lustre-OST0000: new disk, initializing [ 947.508456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 947.511403] Lustre: 78116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 947.537834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 949.454183] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 952.983439] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 953.343041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 953.347520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 953.391193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 954.294648] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 954.360118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 955.650873] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 955.723626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 958.436374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.437357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.437996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.438939] LustreError: Skipped 1 previous similar message [ 958.443714] Lustre: Skipped 1 previous similar message [ 962.449336] Lustre: server umount lustre-OST0000 complete [ 964.064722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 964.067140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.070996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.072718] Lustre: Skipped 1 previous similar message [ 968.673848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.819997] Lustre: server umount lustre-MDT0000 complete [ 971.038193] LustreError: 76143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940511 with bad export cookie 6293341864371717932 [ 971.041856] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.042310] LustreError: 76143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 971.155824] Lustre: server umount lustre-MDT0001 complete [ 973.787238] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 00:21:53 (1786940513) [ 975.843886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.006595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.351792] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 980.026352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.472666] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 983.702751] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.391827] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 987.441451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 989.325285] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 991.621939] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 992.740600] Lustre: 83192:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 1d598eab-230b-4625-8a93-1e6121b50aba at adminstrative request [ 995.811929] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.815253] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1006.052921] Lustre: MGS: Client 1d598eab-230b-4625-8a93-1e6121b50aba (at 0@lo) reconnecting [ 1006.055835] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x5756711503f88a90 to 0x5756711503f88d1b [ 1006.059355] Lustre: MGC192.168.204.124@tcp: Connection restored to 0@lo (at 0@lo) [ 1025.124867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.206144] Lustre: lustre-OST0001: new disk, initializing [ 1025.208297] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1025.210500] Lustre: 83586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1025.243915] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1025.245786] Lustre: Skipped 2 previous similar messages [ 1027.200360] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1030.868349] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1031.828951] Lustre: server umount lustre-OST0001 complete [ 1032.694869] LustreError: 83206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.700183] LustreError: 83206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1036.769522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.770371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.773034] LustreError: Skipped 1 previous similar message [ 1036.778263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.782121] Lustre: Skipped 3 previous similar messages [ 1037.796552] LustreError: 83206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.805396] LustreError: 83206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1039.186397] Lustre: server umount lustre-OST0000 complete [ 1041.888766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1046.612727] Lustre: server umount lustre-MDT0000 complete [ 1047.781279] LustreError: 80506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940588 with bad export cookie 6293341864371719451 [ 1047.784194] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1047.785023] LustreError: 80506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1047.899627] Lustre: server umount lustre-MDT0001 complete [ 1050.326226] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 00:23:09 (1786940589) [ 1050.787576] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1051.350557] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 00:23:10 (1786940590) [ 1051.850773] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1052.431257] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 00:23:11 (1786940591) [ 1052.904742] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1053.444444] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 00:23:12 (1786940592) [ 1053.936705] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 1054.504716] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 00:23:13 (1786940593) [ 1055.125679] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1058.152914] loop: module loaded [ 1058.166404] loop0: detected capacity change from 0 to 4294967296 [ 1058.197180] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1059.149510] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1061.072902] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1063.748691] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1081.592432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1081.639511] LustreError: 87702:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1081.656048] LustreError: 87702:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1081.666289] LustreError: 87702:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1081.680284] LustreError: 87702:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1098.054239] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 1102.577285] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1116.591789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.714090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1116.725923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1116.764131] Lustre: lustre-MDT0000: new disk, initializing [ 1116.795590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.803284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1118.235153] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1122.916509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.974541] Lustre: 90561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1123.106937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1124.820489] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1129.009491] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1131.266131] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1134.288814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1134.415472] Lustre: 92532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1136.778907] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1140.233298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1140.291155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1141.275274] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1143.184823] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1143.305854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1145.202966] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1145.301409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1150.437935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.438554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.439405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.439410] Lustre: Skipped 5 previous similar messages [ 1150.441912] LustreError: Skipped 1 previous similar message [ 1150.446405] Lustre: Skipped 4 previous similar messages [ 1152.280926] Lustre: server umount lustre-OST0000 complete [ 1168.352138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1168.423436] Lustre: server umount lustre-MDT0000 complete [ 1169.828648] LustreError: 90599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940710 with bad export cookie 6293341864371720060 [ 1169.833417] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.835184] LustreError: 90599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1169.971811] Lustre: server umount lustre-MDT0001 complete [ 1173.352772] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 00:25:12 (1786940712) [ 1176.012395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.209855] LustreError: 94939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.749245] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1181.491631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.422765] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1186.621719] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1188.972283] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1192.011749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1194.563370] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1197.715442] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1199.730853] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1201.900367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1203.664324] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1206.855501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 1209.567161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1209.647650] Lustre: lustre-OST0001: new disk, initializing [ 1209.650057] Lustre: Skipped 2 previous similar messages [ 1209.653291] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1209.656167] Lustre: Skipped 3 previous similar messages [ 1209.660558] Lustre: 98342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1212.139814] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1216.527800] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1218.047886] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1218.052846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1218.055783] Lustre: Skipped 2 previous similar messages [ 1218.057919] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1218.083244] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1219.145190] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1220.638737] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1220.721138] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1231.879709] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1240.164381] Lustre: Failing over lustre-MDT0000 [ 1240.367224] Lustre: server umount lustre-MDT0000 complete [ 1242.802965] LustreError: 94935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.812198] LustreError: 94935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1243.112048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1243.118405] LustreError: Skipped 1 previous similar message [ 1243.125733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.147024] Lustre: Skipped 2 previous similar messages [ 1247.896445] LustreError: 98897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.910492] LustreError: 98897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1258.162948] LustreError: 94934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.180480] LustreError: 94934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 1258.977381] Lustre: 66387:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786940783/real 1786940783] req@00000000dded71d4 x1873742803434752/t0(0) o400->MGC192.168.204.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786940799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1259.006401] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1262.589433] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1262.592767] LDISKFS-fs (dm-0): recovery complete [ 1262.602559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.494815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 1269.506363] Lustre: Skipped 6 previous similar messages [ 1269.550606] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1273.497747] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1273.502323] Lustre: 94936:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1273.777544] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1274.859019] Lustre: 94940:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1274.869516] Lustre: 94940:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 1274.885669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1274.997028] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1275.314836] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1275.649757] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1275.665564] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 1275.985992] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1275.995164] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 1276.009110] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1276.020319] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 1276.673668] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1276.680638] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 1277.000462] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1277.015765] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 1278.316433] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1278.336580] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 6 previous similar messages [ 1278.986104] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1279.001465] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 1279.328131] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1279.336940] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 1282.500530] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1282.508295] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 1283.155132] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1283.171188] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 1283.488253] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1283.499087] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 1290.801205] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1290.810032] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 25 previous similar messages [ 1291.441997] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1291.453731] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 25 previous similar messages [ 1291.768154] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1291.774199] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 25 previous similar messages [ 1306.871293] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1306.878964] Lustre: 101173:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 50 previous similar messages [ 1307.506794] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1307.514912] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 50 previous similar messages [ 1307.840158] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1307.844110] LustreError: 101173:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 50 previous similar messages [ 1333.200138] Lustre: 101173:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1333.205219] Lustre: 101173:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1333.209818] Lustre: 101173:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 1829cd8f-ea4a-497e-b010-9eeb47fc496c@192.168.204.24@tcp [ 1333.215992] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1333.218913] Lustre: 101173:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@00000000039e4955 x1873742797017472/t0(8589934781) o101->1829cd8f-ea4a-497e-b010-9eeb47fc496c@192.168.204.24@tcp:89/0 lens 592/0 e 0 to 0 dl 1786940884 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1333.233648] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1333.238248] Lustre: lustre-MDT0000: Denying connection for new client 1829cd8f-ea4a-497e-b010-9eeb47fc496c (at 192.168.204.24@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 1333.241781] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1333.259240] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1333.262484] Lustre: Skipped 2 previous similar messages [ 1333.264496] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1333.283185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 1333.283531] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 1341.411586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.416727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.420150] Lustre: Skipped 4 previous similar messages [ 1341.427629] Lustre: Skipped 9 previous similar messages [ 1347.229313] Lustre: server umount lustre-OST0000 complete [ 1351.139863] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1351.650481] LustreError: 96759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.660368] LustreError: 96759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 1355.502820] Lustre: server umount lustre-OST0001 complete [ 1363.746357] Lustre: server umount lustre-MDT0000 complete [ 1365.712977] LustreError: 96780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940905 with bad export cookie 6293341864371771251 [ 1365.714702] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.720786] LustreError: 96780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1365.887229] Lustre: server umount lustre-MDT0001 complete [ 1369.967592] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 00:28:29 (1786940909) [ 1373.823937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.828465] Lustre: *** cfs_fail_loc=197, val=0*** [ 1389.345676] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1394.464762] LustreError: 103145:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.124@tcp: failed processing log, type 4: rc = -110 [ 1399.620985] Lustre: *** cfs_fail_loc=197, val=0*** [ 1399.628932] Lustre: Skipped 65 previous similar messages [ 1424.642561] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1429.794481] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1430.906403] Lustre: server umount lustre-OST0000 complete [ 1435.341160] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 00:29:34 (1786940974) [ 1452.356685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.616487] LustreError: 104765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.320150] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1460.775673] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.471938] LustreError: 104796:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1463.478258] LustreError: 104796:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1463.484924] LustreError: 104796:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 1463.689550] Lustre: server umount lustre-MDT0000 complete [ 1478.880835] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 1484.055576] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1503.757873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.259328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1504.309353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1504.388613] Lustre: lustre-MDT0000: new disk, initializing [ 1504.494513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1510.875858] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1525.873040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.109573] Lustre: 108962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1526.461539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1526.466129] Lustre: Skipped 3 previous similar messages [ 1526.513216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1533.926444] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1547.985560] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1557.032253] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1563.608304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1564.024942] Lustre: 110932:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1570.523466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1570.612477] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1577.880292] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1581.578704] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1581.755678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1585.346922] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1585.491618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1591.268988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1591.269350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.272294] LustreError: Skipped 2 previous similar messages [ 1591.276692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1591.284528] Lustre: Skipped 5 previous similar messages [ 1591.291612] Lustre: Skipped 7 previous similar messages [ 1592.667162] Lustre: server umount lustre-OST0000 complete [ 1601.214409] Lustre: server umount lustre-MDT0000 complete [ 1601.506655] LustreError: 108972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.525163] LustreError: 108972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1603.736566] LustreError: 108954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941144 with bad export cookie 6293341864371781947 [ 1603.743725] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1603.745433] LustreError: 108954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1603.934622] Lustre: server umount lustre-MDT0001 complete [ 1608.817574] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 00:32:27 (1786941147) [ 1614.187704] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 1616.260610] Key type lgssc unregistered [ 1616.512575] LNet: 113417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.534239] LNetError: 113417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.576606] LNet: Removed LNI 192.168.204.124@tcp [ 1618.060947] Key type .llcrypt unregistered [ 1618.062989] Key type ._llcrypt unregistered [ 1628.605452] Key type ._llcrypt registered [ 1628.607581] Key type .llcrypt registered [ 1628.674260] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 1638.760179] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1639.447396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1639.465254] alg: No test for adler32 (adler32-zlib) [ 1640.391290] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 1640.536051] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 1642.168815] Key type lgssc registered [ 1642.908733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1664.684684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1664.702962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.835251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1665.856325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1665.902947] Lustre: lustre-MDT0000: new disk, initializing [ 1665.939479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.948801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1668.450484] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1675.051725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1675.228759] Lustre: lustre-OST0000: new disk, initializing [ 1675.231674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1675.235819] Lustre: Skipped 1 previous similar message [ 1675.239665] Lustre: 117565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1675.280639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.542564] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1679.231664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1679.239839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1679.261546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1685.584665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1685.699713] Lustre: lustre-OST0001: new disk, initializing [ 1685.702204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1685.706313] Lustre: 118427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1685.750716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.687683] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1691.719803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1691.731561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1691.758334] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1696.738073] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.745102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.751526] Lustre: Skipped 1 previous similar message [ 1701.858549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.864260] Lustre: Skipped 2 previous similar messages [ 1701.971158] Lustre: server umount lustre-MDT0000 complete [ 1706.177892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1709.176771] LustreError: 118950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941249 with bad export cookie 18343306041312716719 [ 1709.185307] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.246483] Lustre: server umount lustre-OST0000 complete [ 1711.861600] Lustre: server umount lustre-OST0001 complete [ 1717.455696] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 00:34:16 (1786941256) [ 1731.678203] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 00:34:30 (1786941270) [ 1738.794675] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1753.078621] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 00:34:51 (1786941291) [ 1777.400676] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 1784.301582] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1806.417985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.586604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1806.605337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1806.655522] Lustre: lustre-MDT0000: new disk, initializing [ 1806.710109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.726881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1809.586298] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1816.835134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.936590] Lustre: 126169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1816.980514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1816.983576] Lustre: Skipped 1 previous similar message [ 1817.036964] Lustre: lustre-MDT0001: new disk, initializing [ 1817.092644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1817.112278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1817.120362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1819.417235] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1824.732168] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1827.457444] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1830.858209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1831.004277] Lustre: lustre-OST0000: new disk, initializing [ 1831.008391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1831.012848] Lustre: 128139:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1831.083295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.950512] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1836.529348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1836.533612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1836.564527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1838.998913] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1841.274309] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1841.425906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1843.416227] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1843.519688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1846.756053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.758454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1846.762936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.771216] Lustre: Skipped 1 previous similar message [ 1850.570377] Lustre: server umount lustre-OST0000 complete [ 1852.897276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1852.901194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.912386] Lustre: Skipped 1 previous similar message [ 1852.918058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.994102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.996229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1857.005423] Lustre: Skipped 1 previous similar message [ 1858.464854] Lustre: server umount lustre-MDT0000 complete [ 1860.045462] LustreError: 126163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941400 with bad export cookie 18343306041312717944 [ 1860.048145] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1860.051756] LustreError: 126163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1860.177135] Lustre: server umount lustre-MDT0001 complete [ 1864.062885] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 00:36:43 (1786941403) [ 1867.246019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.554348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.606606] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1872.866119] LustreError: 130546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.873658] LustreError: 130546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1873.707412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.683768] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1878.728931] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1880.876409] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1883.565429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.192911] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1889.073365] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1896.780516] Lustre: *** cfs_fail_loc=159, val=0*** [ 1896.784212] Lustre: Skipped 1 previous similar message [ 1913.002897] Lustre: lustre-MDT0000: Client 7e288111-847c-498b-ad86-2101da2589df (at 192.168.204.24@tcp) reconnecting [ 1913.614694] Lustre: *** cfs_fail_loc=159, val=0*** [ 1913.616906] Lustre: Skipped 4 previous similar messages [ 1929.905357] Lustre: lustre-MDT0000: Client 7e288111-847c-498b-ad86-2101da2589df (at 192.168.204.24@tcp) reconnecting [ 1946.593631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1946.598296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.605661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.641611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1951.127957] Lustre: server umount lustre-OST0000 complete [ 1964.001582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.004096] Lustre: Skipped 7 previous similar messages [ 1967.072393] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1967.139341] Lustre: server umount lustre-MDT0000 complete [ 1968.976753] LustreError: 130527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941509 with bad export cookie 18343306041312718826 [ 1968.979336] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.983372] LustreError: 130527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1969.146393] Lustre: server umount lustre-MDT0001 complete [ 1974.511195] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 1976.676430] Key type lgssc unregistered [ 1976.857578] LNet: 134323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.865240] LNetError: 134323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.882576] LNet: Removed LNI 192.168.204.124@tcp [ 1977.860901] Key type .llcrypt unregistered [ 1977.867067] Key type ._llcrypt unregistered [ 2012.689994] Key type ._llcrypt registered [ 2012.693690] Key type .llcrypt registered [ 2012.909463] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2015.731954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2016.409018] alg: No test for adler32 (adler32-zlib) [ 2018.437700] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 2019.089374] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2021.066384] Key type lgssc registered [ 2024.504640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2039.293616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2039.374809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.438085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.706540] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2051.554463] LustreError: 135743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.577745] LustreError: 135743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2056.677587] LustreError: 135742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.587935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.815021] LustreError: 135743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.277050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.005838] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2080.436468] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2089.601425] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2098.585894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2098.931308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2105.027523] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2111.238853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2112.523722] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2120.824593] Lustre: *** cfs_fail_loc=159, val=0*** [ 2137.274142] Lustre: lustre-MDT0000: Client 3eedafc4-3319-4a59-8fa1-66bbbd4621a8 (at 192.168.204.24@tcp) reconnecting [ 2139.662446] Lustre: *** cfs_fail_loc=159, val=0*** [ 2156.242610] Lustre: lustre-MDT0001: Client 3eedafc4-3319-4a59-8fa1-66bbbd4621a8 (at 192.168.204.24@tcp) reconnecting [ 2157.674785] Lustre: *** cfs_fail_loc=159, val=0*** [ 2157.689081] Lustre: Skipped 6 previous similar messages [ 2174.185657] Lustre: lustre-MDT0001: Client 3eedafc4-3319-4a59-8fa1-66bbbd4621a8 (at 192.168.204.24@tcp) reconnecting [ 2179.620841] Lustre: 138788:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2182.308857] Lustre: *** cfs_fail_loc=159, val=0*** [ 2182.310741] Lustre: Skipped 1 previous similar message [ 2198.750718] Lustre: lustre-MDT0000: Client 03f82ce4-2deb-41a3-b0e3-6c9b39e9f0a3 (at 192.168.204.24@tcp) reconnecting [ 2200.409066] Lustre: *** cfs_fail_loc=159, val=0*** [ 2200.417111] Lustre: Skipped 14 previous similar messages [ 2216.669435] Lustre: lustre-MDT0000: Client 03f82ce4-2deb-41a3-b0e3-6c9b39e9f0a3 (at 192.168.204.24@tcp) reconnecting [ 2217.901468] Lustre: 139088:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2223.082624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2223.089279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.111335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2224.110703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2224.119227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.400754] Lustre: server umount lustre-OST0000 complete [ 2229.220290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.231727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2229.240611] Lustre: Skipped 2 previous similar messages [ 2234.340497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2234.343974] Lustre: Skipped 1 previous similar message [ 2234.450348] Lustre: server umount lustre-MDT0000 complete [ 2237.673735] LustreError: 138915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941777 with bad export cookie 14657003733082641123 [ 2237.678140] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2237.691061] LustreError: 138915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2237.940893] Lustre: server umount lustre-MDT0001 complete [ 2248.772675] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2251.717094] Key type lgssc unregistered [ 2252.020345] LNet: 140141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.025535] LNetError: 140141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.048879] LNet: Removed LNI 192.168.204.124@tcp [ 2252.952192] Key type .llcrypt unregistered [ 2252.958087] Key type ._llcrypt unregistered [ 2276.296753] Key type ._llcrypt registered [ 2276.299100] Key type .llcrypt registered [ 2276.382688] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2277.331661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2277.354769] alg: No test for adler32 (adler32-zlib) [ 2278.402170] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 2278.613033] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2280.328462] Key type lgssc registered [ 2281.251488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.013751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.043987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.582742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.015254] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2299.942634] LustreError: 141562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.957906] LustreError: 141562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2303.776784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.659377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.864620] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2319.820569] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2327.272675] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2334.534782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2334.977346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.022223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2339.758261] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2347.022230] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2349.364109] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2355.920564] Lustre: 144147:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2359.041627] Lustre: 144196:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2360.345282] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2361.240639] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 2364.550387] Lustre: 144342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2365.390373] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2366.229699] Lustre: 144440:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2372.069040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.081857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.107758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2375.142295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2375.155321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2375.161073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.369538] Lustre: server umount lustre-OST0000 complete [ 2380.261314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.279650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.286543] Lustre: Skipped 1 previous similar message [ 2385.378761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2385.384379] Lustre: Skipped 1 previous similar message [ 2385.834940] Lustre: server umount lustre-MDT0000 complete [ 2389.184033] LustreError: 144159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941929 with bad export cookie 991487051801536709 [ 2389.186164] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.192848] LustreError: 144159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2389.558181] Lustre: server umount lustre-MDT0001 complete [ 2400.716553] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2403.427549] Key type lgssc unregistered [ 2403.674246] LNet: 145504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.680975] LNetError: 145504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2404.718536] LNet: Removed LNI 192.168.204.124@tcp [ 2405.428509] Key type .llcrypt unregistered [ 2405.430807] Key type ._llcrypt unregistered [ 2424.641597] Key type ._llcrypt registered [ 2424.644124] Key type .llcrypt registered [ 2424.773751] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2425.734577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.752555] alg: No test for adler32 (adler32-zlib) [ 2426.786076] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 2427.006292] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2428.696208] Key type lgssc registered [ 2429.585836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2436.312986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2436.335515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.751927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.347247] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2447.331960] LustreError: 146899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.349758] LustreError: 146899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2448.294521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.701104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2452.409126] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2458.677754] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2462.925878] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2468.128099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2468.437169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.854288] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2473.578627] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2478.586753] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2481.778239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2488.596623] Lustre: *** cfs_fail_loc=159, val=0*** [ 2504.898572] Lustre: lustre-MDT0000: Client 4d80bb33-cf6a-4357-be58-ab4705579e98 (at 192.168.204.24@tcp) reconnecting [ 2523.110390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.111192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2523.121505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2523.127427] Lustre: Skipped 1 previous similar message [ 2526.722589] Lustre: server umount lustre-OST0000 complete [ 2530.786519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2530.795249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.808630] Lustre: Skipped 1 previous similar message [ 2530.813396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.349659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.350857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.372728] Lustre: Skipped 1 previous similar message [ 2535.534759] Lustre: server umount lustre-MDT0000 complete [ 2538.416590] LustreError: 146878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942078 with bad export cookie 14843879941801334538 [ 2538.424391] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.425826] LustreError: 146878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2538.690072] Lustre: server umount lustre-MDT0001 complete [ 2546.065569] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2548.061514] Key type lgssc unregistered [ 2548.289825] LNet: 150568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.302327] LNetError: 150568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.319681] LNet: Removed LNI 192.168.204.124@tcp [ 2548.958693] Key type .llcrypt unregistered [ 2548.962139] Key type ._llcrypt unregistered [ 2566.300667] Key type ._llcrypt registered [ 2566.302767] Key type .llcrypt registered [ 2566.395108] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2567.054297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2567.074358] alg: No test for adler32 (adler32-zlib) [ 2568.059297] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 2568.256116] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2569.904216] Key type lgssc registered [ 2570.841222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2577.050089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2577.069700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.418580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.447649] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2587.700677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.017987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2591.306379] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2596.672497] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2600.423695] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2605.255229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2605.516730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.998662] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2613.751899] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2615.226219] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2623.985636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2642.280436] Lustre: lustre-MDT0001: evicting df6a0b31-a5aa-4816-8a0b-5204961bb08c (at 192.168.204.24@tcp) by administrative request [ 2642.286659] Lustre: Skipped 1 previous similar message [ 2670.050934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2670.055341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2670.066066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.174596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.183097] Lustre: Skipped 1 previous similar message [ 2680.292888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2680.297973] Lustre: Skipped 1 previous similar message [ 2682.336122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2682.420918] Lustre: server umount lustre-OST0000 complete [ 2685.411937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.413182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2685.417654] Lustre: Skipped 1 previous similar message [ 2685.421328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.423601] LustreError: Skipped 1 previous similar message [ 2685.444224] Lustre: Skipped 2 previous similar messages [ 2690.530651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.537952] Lustre: Skipped 1 previous similar message [ 2699.232166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2699.317673] Lustre: server umount lustre-MDT0000 complete [ 2700.769687] LustreError: 153702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.778233] LustreError: 153702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2701.563227] LustreError: 151943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942241 with bad export cookie 14314846417598054266 [ 2701.568544] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2701.569045] LustreError: 151943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2701.769054] Lustre: server umount lustre-MDT0001 complete [ 2708.426360] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2710.235580] Key type lgssc unregistered [ 2710.435512] LNet: 156053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2710.439051] LNetError: 156053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2710.455541] LNet: Removed LNI 192.168.204.124@tcp [ 2710.982493] Key type .llcrypt unregistered [ 2710.984397] Key type ._llcrypt unregistered [ 2757.084947] Key type ._llcrypt registered [ 2757.086821] Key type .llcrypt registered [ 2757.156668] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2757.750460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2757.778504] alg: No test for adler32 (adler32-zlib) [ 2758.700837] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 2758.843881] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2760.472204] Key type lgssc registered [ 2761.183673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2766.112199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2766.143100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.470405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.189523] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2775.606176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.857130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.416625] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2782.482321] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2785.609888] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2789.236876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.431272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.492281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2792.574383] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2796.465934] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2801.710174] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2806.753546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2806.757603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.762392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2806.768419] Lustre: Skipped 1 previous similar message [ 2806.775438] Lustre: Skipped 1 previous similar message [ 2808.368420] Lustre: server umount lustre-OST0000 complete [ 2811.876114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2811.878943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.884096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.890133] Lustre: Skipped 1 previous similar message [ 2811.899363] Lustre: Skipped 1 previous similar message [ 2816.699029] Lustre: server umount lustre-MDT0000 complete [ 2816.995310] LustreError: 159246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.006262] LustreError: 159246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2819.179778] LustreError: 158495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942359 with bad export cookie 3853151203199187503 [ 2819.180488] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2819.189420] LustreError: 158495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2819.425932] Lustre: server umount lustre-MDT0001 complete [ 2826.805792] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2828.858190] Key type lgssc unregistered [ 2829.079347] LNet: 162056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2829.085841] LNetError: 162056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2829.096092] LNet: Removed LNI 192.168.204.124@tcp [ 2829.708695] Key type .llcrypt unregistered [ 2829.712706] Key type ._llcrypt unregistered [ 2847.447018] Key type ._llcrypt registered [ 2847.448811] Key type .llcrypt registered [ 2847.519777] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2848.214484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2848.240306] alg: No test for adler32 (adler32-zlib) [ 2849.253902] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 2849.427146] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2851.056196] Key type lgssc registered [ 2851.846423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2861.807230] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 00:53:20 (1786942400) [ 2873.200489] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 2879.545379] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2903.018718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2903.038076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.257556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2904.288083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2904.369848] Lustre: lustre-MDT0000: new disk, initializing [ 2904.450195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.465522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2908.159252] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2916.549972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.642881] Lustre: 166294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2916.704584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2916.709774] Lustre: Skipped 1 previous similar message [ 2916.801244] Lustre: lustre-MDT0001: new disk, initializing [ 2916.831550] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2916.921958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2916.974906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2916.985450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2920.205948] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2927.527916] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2931.808561] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2936.957382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2937.142197] Lustre: lustre-OST0000: new disk, initializing [ 2937.145848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2937.150928] Lustre: 168262:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2937.222320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.885086] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2943.555368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2943.565946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2943.668163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2949.537423] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2957.555220] Lustre: server umount lustre-OST0000 complete [ 2962.913526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2962.917139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.925100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.450965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.451510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.570763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.575076] Lustre: Skipped 2 previous similar messages [ 2974.176282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2974.302650] Lustre: server umount lustre-MDT0000 complete [ 2975.586531] Lustre: 162760:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786942499/real 1786942499] req@00000000a315722b x1873744902520704/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786942515 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2975.597593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.540933] LustreError: 166287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942516 with bad export cookie 4388562640358835350 [ 2976.549317] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.550920] LustreError: 166287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2976.722452] Lustre: server umount lustre-MDT0001 complete [ 2982.746113] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2984.529434] Key type lgssc unregistered [ 2984.741193] LNet: 170175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2984.749588] LNetError: 170175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.768261] LNet: Removed LNI 192.168.204.124@tcp [ 2985.390746] Key type .llcrypt unregistered [ 2985.395525] Key type ._llcrypt unregistered [ 3000.744448] Key type ._llcrypt registered [ 3000.746508] Key type .llcrypt registered [ 3000.802304] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3001.286589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3001.387128] alg: No test for adler32 (adler32-zlib) [ 3002.333368] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 3002.506694] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 3004.146584] Key type lgssc registered [ 3004.841154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3009.049602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3009.070494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.335428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.555437] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3017.391824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.618216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.135487] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3024.811777] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3028.248228] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3032.530456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3032.757229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3034.801763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3037.009326] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3041.473333] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3045.768907] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3045.782150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.796690] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3045.800656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3045.808532] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3046.343234] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3046.355826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.362586] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3046.368393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3046.376614] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3046.993538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3047.561437] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3047.565401] Lustre: Skipped 1 previous similar message [ 3047.579785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.589290] Lustre: Skipped 1 previous similar message [ 3047.593095] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3047.599702] Lustre: Skipped 1 previous similar message [ 3047.603141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3047.613313] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3047.619740] Lustre: Skipped 1 previous similar message [ 3048.187457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3048.680478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3049.220496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3049.782459] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3049.789352] Lustre: Skipped 3 previous similar messages [ 3049.800732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.809380] Lustre: Skipped 3 previous similar messages [ 3049.813754] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3049.818469] Lustre: Skipped 3 previous similar messages [ 3049.823087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3049.829553] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3049.832700] Lustre: Skipped 3 previous similar messages [ 3050.335843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3050.905595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3051.484717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3052.106351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3052.657655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3053.206363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3053.767904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3054.304275] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3054.313015] Lustre: Skipped 7 previous similar messages [ 3054.334675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.349701] Lustre: Skipped 7 previous similar messages [ 3054.356988] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3054.362820] Lustre: Skipped 7 previous similar messages [ 3054.366464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3054.374582] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3054.380612] Lustre: Skipped 7 previous similar messages [ 3054.959957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3055.498276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3055.976577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3056.517848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3057.125693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3057.712341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3058.244933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3058.787637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3059.420045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3059.955973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3060.412685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3060.896963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3061.406622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3061.864299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3062.357861] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3062.361414] Lustre: Skipped 14 previous similar messages [ 3062.369590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.375887] Lustre: Skipped 14 previous similar messages [ 3062.378646] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3062.382990] Lustre: Skipped 14 previous similar messages [ 3062.386317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3062.391037] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3062.393723] Lustre: Skipped 14 previous similar messages [ 3062.896343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3063.392251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3063.858414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3064.375143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3064.872290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3065.392974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3065.930924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3066.434972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3066.961932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3067.508106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3068.012719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3068.498824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3069.001682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3069.521969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3070.032189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3070.540152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3071.051103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3071.537157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3072.018245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3072.648404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3073.190647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3073.759408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3074.303775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3074.830025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3075.355389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3075.881497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3076.379333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3076.902857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3077.417873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3077.980105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3078.478940] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3078.483126] Lustre: Skipped 30 previous similar messages [ 3078.495737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.502674] Lustre: Skipped 30 previous similar messages [ 3078.508057] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3078.513535] Lustre: Skipped 30 previous similar messages [ 3078.517290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3078.529204] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3078.535817] Lustre: Skipped 30 previous similar messages [ 3079.052153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3079.563186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3080.076065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3080.663344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3081.147393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3081.678693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3082.224235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3082.786255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3083.352636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3083.847980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3084.314242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3084.817536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3085.297722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3085.783645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3086.297481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3086.824645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3087.322067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3087.831931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3088.326441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3088.793930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3089.247871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3089.745789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3090.211449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3090.659367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3091.155599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3091.644845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3092.057555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3092.493768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3092.947026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3093.411466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3093.829864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3094.268861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3094.679286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3095.128490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3095.546334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3095.975828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3096.449291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3096.919357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3097.381922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3097.882201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3098.368400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3098.837562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3099.335608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3099.850463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3100.358277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3100.847417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3101.324715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3101.814736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3102.340590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3102.828021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3103.353723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3103.848846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3104.385866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3104.934854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3105.501135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3106.070361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3106.687440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3107.250838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3107.906860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3108.480056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3109.001887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3109.480612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3109.940818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3110.462533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3110.898221] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3110.902071] Lustre: Skipped 64 previous similar messages [ 3110.914344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.921796] Lustre: Skipped 64 previous similar messages [ 3110.926411] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3110.929773] Lustre: Skipped 64 previous similar messages [ 3110.933255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3110.940662] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3110.943672] Lustre: Skipped 64 previous similar messages [ 3111.430984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3111.894680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3112.359741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3112.873049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3113.367958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3113.866401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3114.353019] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3114.822823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3115.314187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3115.791995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3116.263563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3116.730638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3117.208344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3117.675737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3118.167343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3118.633693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3119.079384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3119.556753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3120.098650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3120.558123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3121.048459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3121.559700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3122.057410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3122.541234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3122.986529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3123.503367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3123.978276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3124.428667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3124.919574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3125.418574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3125.918814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3126.474511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3126.984653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3127.487777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3128.049083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3128.555684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3129.029896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3129.560703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3130.045919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3130.550116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3131.093219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3131.636902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3132.191963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3132.673249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3133.173271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3133.731163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3134.267464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3134.744660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3135.228879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3135.718058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3136.186947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3136.688889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3137.210484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3137.696571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3138.181872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3138.739620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3139.265983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3139.745640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3140.251266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3140.709387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3141.203636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3141.784336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3142.279463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3142.829500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3143.296920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3143.790958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3144.319319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3144.822900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3145.332300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3145.853191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3146.353785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3146.857180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3147.383952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3147.890349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3148.405453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3148.938630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3149.490345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3150.021036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3150.543840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3151.146483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3151.675508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3152.224789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3152.771947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3153.300881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3153.837107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3154.335407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3154.862451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3155.385454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3155.926610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3156.427453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3156.934724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3157.421749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3157.965467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3158.486362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3158.996452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3159.536564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3160.063550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3160.548935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3161.034525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3161.555440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3162.071082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3162.568933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3163.114588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3163.639299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3164.104292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3164.623077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3165.085518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3165.548990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3166.027804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3166.491787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3166.978421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3167.454686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3167.940476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3168.366092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3168.827987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3169.303691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3169.775931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3170.298456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3170.762815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3171.221114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3171.685201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3172.149198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3172.586493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3173.056611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3173.478165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3173.920296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3174.452971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3174.970677] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3174.976967] Lustre: Skipped 127 previous similar messages [ 3174.989314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.002757] Lustre: Skipped 127 previous similar messages [ 3175.005636] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3175.013062] Lustre: Skipped 127 previous similar messages [ 3175.016065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3175.030651] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3175.038288] Lustre: Skipped 127 previous similar messages [ 3175.533409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3176.030197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3176.494811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3177.007292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3177.453574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3177.972475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3178.466982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3178.943255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3179.407353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3179.987505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3180.488080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3181.011151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3181.587453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3182.169728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3182.749807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3183.316705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3183.821582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3184.363167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3184.876729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3185.460611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3186.064540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3186.567218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3187.103510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3187.608581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3188.125893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3188.670683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3189.190375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3189.674317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3190.196766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3190.644353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3191.131497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3191.602064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3192.097752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3192.580601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3193.038739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3193.548480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3194.043777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3194.515922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3195.038609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3195.551066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3196.038149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3196.507643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3196.987412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3197.496850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3197.945593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3198.365587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3198.825409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3199.293477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3199.744662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3200.234331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3200.683487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3201.121837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3201.593279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3202.053811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3202.640924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3203.157149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3203.640216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3204.120608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3204.603674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3205.093092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3205.570610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3206.048370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3206.540871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3206.999409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3207.476979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3208.037093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3208.521602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3209.044405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3209.521526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3210.033894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3210.485807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3211.009606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3211.478951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3211.920572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3212.378233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3212.842584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3213.277031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3213.723719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3214.191241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3214.631093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3215.128657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3215.635155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3216.133794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3216.616573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3217.187977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3217.672575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3218.192487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3218.694527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3219.182041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3219.634485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3220.110938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3220.565334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3221.014130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3221.499083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3221.965695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3222.430782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3222.934274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3223.398823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3223.863649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3224.339614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3224.830559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3225.323330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3225.851391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3226.354377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3226.843026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3227.427909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3227.961165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3228.443941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3228.985397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3229.493478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3230.004574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3230.457895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3230.966289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3231.460884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3231.976797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3232.524079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3233.051308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3233.570881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3234.083916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3234.627162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3235.164317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3235.636338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3236.201391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3236.691896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3237.143340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3237.600498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3238.038240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3238.490397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3239.024935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3239.520524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3240.013851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3240.491919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3240.968278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3241.486249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3241.989642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3242.486308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3242.957634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3243.467992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3243.944739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3244.437178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3245.004393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3245.455609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3245.935323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3246.428568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3246.903826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3247.415413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3247.907626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3248.343540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3248.804801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3249.250308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3249.717694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3250.194300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3250.660536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3251.066287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3251.569557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3252.086563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3252.539903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3252.999541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3253.508204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3253.989123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3254.489267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3254.991660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3255.428321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3255.877554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3256.284899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3256.732518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3257.148639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3257.550117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3257.963299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3258.353208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3258.749310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3259.182219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3259.567368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3259.978050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3260.395439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3260.836792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3261.250469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3261.680196] LustreError: 171586:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 3261.687083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3261.692608] LustreError: 171586:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 3262.162442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3262.635636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3263.032367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3263.475912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3263.946337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3264.376764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3264.828080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3292.141771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3292.144961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3292.150091] LustreError: Skipped 4 previous similar messages [ 3293.154453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3293.160338] LustreError: Skipped 3 previous similar messages [ 3293.164208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.356483] Lustre: server umount lustre-OST0000 complete [ 3296.227683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3296.231495] Lustre: Skipped 1 previous similar message [ 3301.345407] LustreError: 171562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.353865] LustreError: 171562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3301.373315] Lustre: server umount lustre-MDT0000 complete [ 3303.050436] LustreError: 171548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942843 with bad export cookie 1169327063699322235 [ 3303.055870] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3303.056832] LustreError: 171548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3303.213420] Lustre: server umount lustre-MDT0001 complete [ 3308.175996] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 3309.625165] Key type lgssc unregistered [ 3309.784811] LNet: 197951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3309.788429] LNetError: 197951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3309.803087] LNet: Removed LNI 192.168.204.124@tcp [ 3310.285790] Key type .llcrypt unregistered [ 3310.287342] Key type ._llcrypt unregistered [ 3323.454386] Key type ._llcrypt registered [ 3323.456359] Key type .llcrypt registered [ 3323.527673] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3324.124539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.141550] alg: No test for adler32 (adler32-zlib) [ 3325.083143] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 3325.214694] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 3326.832184] Key type lgssc registered [ 3327.446920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3331.423804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3331.434239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.654886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.675407] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3338.972559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.169743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3341.686800] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3349.685796] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3358.202875] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3369.206710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.613987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.931739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5811 to 0x280000401:5857) [ 3380.398125] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3397.745501] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3405.833079] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3409.933441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.957982] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3409.981664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3410.009156] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3410.011130] LustreError: 199375:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 3415.011821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.028098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.032222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.065409] Lustre: Skipped 1 previous similar message [ 3416.269963] Lustre: lustre-OST0000: Not available for connect from 192.168.204.24@tcp (stopping) [ 3416.277621] Lustre: Skipped 1 previous similar message [ 3419.636665] Lustre: server umount lustre-OST0000 complete [ 3430.072109] LustreError: 199375:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 3453.457722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3453.801655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.826391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3454.845882] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3454.853807] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3454.870256] Lustre: Skipped 1 previous similar message [ 3459.909257] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3466.700439] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3467.977345] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3472.753417] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff939445174800.ost_server_uuid 50 [ 3474.295172] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff939445174800.ost_server_uuid in FULL state after 0 sec [ 3479.521132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3479.530543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.546354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3480.547743] Lustre: server umount lustre-OST0000 complete [ 3485.666391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.671802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.686597] Lustre: Skipped 1 previous similar message [ 3490.236862] Lustre: server umount lustre-MDT0000 complete [ 3490.785304] LustreError: 200082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.804749] LustreError: 200082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3493.632095] LustreError: 199337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786943033 with bad export cookie 6541871122545926781 [ 3493.642570] LustreError: 199337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3493.648150] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3503.225650] Lustre: server umount lustre-MDT0001 complete [ 3514.137561] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 3517.112987] Key type lgssc unregistered [ 3517.499778] LNet: 204569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3517.505295] LNetError: 204569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3517.542533] LNet: Removed LNI 192.168.204.124@tcp [ 3518.507770] Key type .llcrypt unregistered [ 3518.512880] Key type ._llcrypt unregistered [ 3548.342787] Key type ._llcrypt registered [ 3548.344754] Key type .llcrypt registered [ 3548.460577] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3549.574899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3549.632415] alg: No test for adler32 (adler32-zlib) [ 3550.708822] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 3550.964047] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 3552.672398] Key type lgssc registered [ 3553.931817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3562.726642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3562.755503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.144530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3567.977778] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3573.735162] LustreError: 206299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.753072] LustreError: 206299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3575.595117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.975971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3579.796438] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3582.138372] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3587.388498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3587.895122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.926937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5859 to 0x280000401:5889) [ 3593.529599] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3600.799647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3601.060347] Lustre: lustre-OST0001: new disk, initializing [ 3601.067092] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3601.077143] Lustre: 208606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3601.180416] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3607.394763] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3610.264750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3610.312985] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3610.476887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3617.374073] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3624.994886] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3637.217542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3637.238818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.270865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3641.318343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3641.322861] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.352873] Lustre: Skipped 2 previous similar messages [ 3643.371807] Lustre: server umount lustre-MDT0000 complete [ 3646.444533] LustreError: 207805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.481785] LustreError: 207805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3647.688769] LustreError: 209548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786943187 with bad export cookie 17910379255822271736 [ 3647.698579] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3647.711398] LustreError: 209548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3648.453118] Lustre: server umount lustre-MDT0001 complete [ 3663.169670] Lustre: server umount lustre-OST0000 complete [ 3677.636614] Lustre: server umount lustre-OST0001 complete [ 3699.509278] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3709.532481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.079140] LustreError: 211713:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.174860] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.477604] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3715.557884] LustreError: 211714:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.693409] LustreError: 211713:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.870523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.277610] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3730.534378] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3733.844365] Lustre: 212854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3741.579709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3742.027711] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.387933] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3750.190904] LustreError: 213210:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3755.326121] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5891 to 0x280000401:5921) [ 3757.847611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3763.216298] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3765.844581] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3774.046469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3777.586787] Lustre: 214725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3788.775756] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.780674] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3788.809746] Lustre: Skipped 2 previous similar messages [ 3788.821481] Lustre: Skipped 6 previous similar messages [ 3793.907264] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3794.633186] Lustre: server umount mylustre-MDT0000 complete [ 3798.163020] LustreError: 211690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786943338 with bad export cookie 17910379255822274536 [ 3798.177119] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3798.179620] LustreError: 211690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3798.566909] Lustre: server umount mylustre-MDT0001 complete [ 3811.952382] Lustre: server umount mylustre-OST0000 complete [ 3814.881158] Lustre: 205562:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786943339/real 1786943339] req@ffff9980829c0380 x1873745638739456/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786943355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3814.908356] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3814.918700] Lustre: Skipped 1 previous similar message [ 3815.743448] Lustre: server umount mylustre-OST0001 complete [ 3837.449771] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3847.098621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.461927] LustreError: 217034:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3847.486947] LustreError: 217034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3847.550319] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3847.554311] Lustre: Skipped 1 previous similar message [ 3851.334935] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3858.933369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.162156] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3865.713288] Lustre: 218177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3871.255639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3872.501161] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5923 to 0x280000401:5953) [ 3876.884207] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3882.977427] LustreError: 218529:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.996310] LustreError: 218529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3884.554798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3890.194365] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 3890.907774] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3897.328755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3900.313466] Lustre: 220050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3910.634374] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.637787] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3910.638986] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.638992] Lustre: Skipped 3 previous similar messages [ 3910.655079] Lustre: Skipped 1 previous similar message [ 3912.222094] Lustre: server umount tfs-MDT0000 complete [ 3915.161389] LustreError: 217011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786943455 with bad export cookie 17910379255822276426 [ 3915.163282] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3915.171813] LustreError: 217011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3915.489351] Lustre: server umount tfs-MDT0001 complete [ 3929.063963] Lustre: server umount tfs-OST0000 complete [ 3942.317566] Lustre: server umount tfs-OST0001 complete [ 3959.453608] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3968.126395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3968.512906] LustreError: 222300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.533534] LustreError: 222300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3968.609434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3968.618678] Lustre: Skipped 3 previous similar messages [ 3972.686320] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3980.678105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.577852] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3988.280075] Lustre: 223443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3994.430808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3999.730359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5923 to 0x280000401:5985) [ 3999.980189] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4007.094100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.542293] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 4012.728193] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4019.080181] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4021.770891] Lustre: 225311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4027.389484] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 01:12:45 (1786943565) [ 4042.722333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4042.730224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.739205] Lustre: Skipped 2 previous similar messages [ 4042.745559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4042.749882] Lustre: Skipped 3 previous similar messages [ 4048.091194] Lustre: server umount lustre-MDT0000 complete [ 4050.914964] LustreError: 223445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786943591 with bad export cookie 17910379255822278092 [ 4050.920963] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4050.927293] LustreError: 223445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4060.641370] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4060.645949] Lustre: Skipped 9 previous similar messages [ 4065.251389] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4065.499317] Lustre: server umount lustre-MDT0001 complete [ 4075.197471] Lustre: server umount lustre-OST0000 complete [ 4084.425350] Lustre: server umount lustre-OST0001 complete [ 4089.563166] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 4097.222191] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4124.546172] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 4124.546172] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4124.546172] [ 4124.572645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4124.720506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4124.754817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4124.884883] Lustre: lustre-MDT0000: new disk, initializing [ 4125.002212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.018261] Lustre: Skipped 3 previous similar messages [ 4125.030657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4128.367435] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4136.701474] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 4136.701474] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4136.701474] [ 4136.734033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4136.816304] Lustre: 228502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4136.864068] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4136.867527] Lustre: Skipped 1 previous similar message [ 4136.944352] Lustre: lustre-MDT0001: new disk, initializing [ 4137.076867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4137.084256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4140.358267] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4148.609393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4148.825846] Lustre: lustre-OST0000: new disk, initializing [ 4148.831532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4148.843572] Lustre: 230081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4149.962142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4149.972599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4150.055203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4153.697715] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4161.597192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4161.735918] Lustre: lustre-OST0001: new disk, initializing [ 4161.743332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4161.750540] Lustre: 230953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4162.872351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4162.878472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4162.964118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4166.230239] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4174.748226] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 01:15:13 (1786943713) [ 4184.042057] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 01:15:22 (1786943722) [ 4198.895463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.904686] Lustre: Skipped 5 previous similar messages [ 4198.909153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.913410] Lustre: Skipped 3 previous similar messages [ 4203.477481] Lustre: server umount lustre-OST0000 complete [ 4204.010338] LustreError: 230171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.027155] LustreError: 230171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 4220.386783] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4220.547155] Lustre: server umount lustre-MDT0000 complete [ 4223.190571] LustreError: 228493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786943763 with bad export cookie 17910379255822279646 [ 4223.194921] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.201804] LustreError: 228493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4229.587268] Lustre: server umount lustre-MDT0001 complete [ 4237.503166] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 4310.860636] Lustre: server umount lustre-OST0001 complete [ 4315.310869] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 4325.633811] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4350.771133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.002241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4351.042890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4351.122667] Lustre: lustre-MDT0000: new disk, initializing [ 4351.215664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4354.729577] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4363.151735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.269239] Lustre: 235726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4363.497176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4366.769323] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4373.620758] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4377.627220] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4382.520270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.732641] Lustre: lustre-OST0000: new disk, initializing [ 4382.734727] Lustre: Skipped 1 previous similar message [ 4382.737988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4382.741823] Lustre: Skipped 2 previous similar messages [ 4382.745885] Lustre: 237698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4382.835948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.843616] Lustre: Skipped 5 previous similar messages [ 4384.163754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4384.175404] Lustre: Skipped 1 previous similar message [ 4384.178376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4384.244108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4386.646151] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4393.111451] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4399.588048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4399.589526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.596115] LustreError: Skipped 1 previous similar message [ 4399.601743] Lustre: Skipped 5 previous similar messages [ 4399.604716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4399.610939] Lustre: Skipped 12 previous similar messages [ 4403.076854] Lustre: server umount lustre-OST0000 complete [ 4409.832379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4411.206513] Lustre: server umount lustre-MDT0000 complete [ 4413.632912] LustreError: 235721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786943953 with bad export cookie 17910379255822281116 [ 4413.639644] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4413.642357] LustreError: 235721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4413.882537] Lustre: server umount lustre-MDT0001 complete [ 4420.296443] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 4422.250023] Key type lgssc unregistered [ 4422.474346] LNet: 239460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4422.484923] LNetError: 239460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4422.498800] LNet: Removed LNI 192.168.204.124@tcp [ 4423.095176] Key type .llcrypt unregistered [ 4423.099839] Key type ._llcrypt unregistered [ 4440.290490] Key type ._llcrypt registered [ 4440.295165] Key type .llcrypt registered [ 4440.365771] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4440.862823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4440.895355] alg: No test for adler32 (adler32-zlib) [ 4441.805064] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 4441.943243] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 4443.552513] Key type lgssc registered [ 4444.183074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4448.774965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4448.788837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.028426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.449947] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4457.292694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.533960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4460.205921] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4465.010177] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4469.104425] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4473.544400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4473.737205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.800758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4477.884154] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4482.424513] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4482.995201] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 4485.090899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4485.096574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.105643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4487.137515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4487.141912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4487.142493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.309509] Lustre: server umount lustre-OST0000 complete [ 4493.282940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4493.289513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.301672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4493.311465] Lustre: Skipped 1 previous similar message [ 4497.377508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4497.378093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.919720] Lustre: server umount lustre-MDT0000 complete [ 4501.234579] LustreError: 242742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786944041 with bad export cookie 11306508293790098895 [ 4501.239789] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.242776] LustreError: 242742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4501.463780] Lustre: server umount lustre-MDT0001 complete [ 4508.177618] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 4509.891386] Key type lgssc unregistered [ 4510.073762] LNet: 244484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4510.084605] LNetError: 244484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4510.102306] LNet: Removed LNI 192.168.204.124@tcp [ 4510.662724] Key type .llcrypt unregistered [ 4510.664644] Key type ._llcrypt unregistered [ 4521.563603] Key type ._llcrypt registered [ 4521.567697] Key type .llcrypt registered [ 4521.630517] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4522.266473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4522.279139] alg: No test for adler32 (adler32-zlib) [ 4523.225151] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 4523.382963] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 4525.048187] Key type lgssc registered [ 4525.791679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4531.058142] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4531.074736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4546.528516] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4551.648602] LustreError: 245657:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.124@tcp: failed processing log, type 4: rc = -110 [ 4578.272566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.002709] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4586.692989] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4588.036962] Lustre: server umount lustre-OST0000 complete [ 4597.864437] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 4599.678848] Key type lgssc unregistered [ 4599.880342] LNet: 247290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4599.889392] LNetError: 247290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4599.899091] LNet: Removed LNI 192.168.204.124@tcp [ 4600.436309] Key type .llcrypt unregistered [ 4600.440417] Key type ._llcrypt unregistered [ 4623.448554] Key type ._llcrypt registered [ 4623.450492] Key type .llcrypt registered [ 4623.522816] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4624.134696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4624.150991] alg: No test for adler32 (adler32-zlib) [ 4625.114130] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 4625.272789] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 4626.888209] Key type lgssc registered [ 4627.548657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4634.747976] loop0: detected capacity change from 0 to 204800000 [ 4634.846471] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.854304] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4635.042042] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4635.544691] loop0: detected capacity change from 0 to 204800000 [ 4635.615968] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4635.622906] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4635.807229] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4636.323955] loop0: detected capacity change from 0 to 204800000 [ 4636.411855] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4636.416949] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4636.478224] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4637.080878] loop0: detected capacity change from 0 to 204800000 [ 4637.205700] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4637.800094] loop0: detected capacity change from 0 to 204800000 [ 4637.818265] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4640.170839] loop0: detected capacity change from 0 to 204800000 [ 4640.182578] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4642.413239] loop0: detected capacity change from 0 to 204800000 [ 4642.430232] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4644.852892] loop0: detected capacity change from 0 to 204800000 [ 4644.881031] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4647.199526] loop0: detected capacity change from 0 to 204800000 [ 4647.254214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4647.275614] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.323784] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4647.330286] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 4647.339939] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 4647.347330] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 4647.355410] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 4647.368583] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 4647.377610] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 4647.386609] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 4647.394148] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 4647.401256] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 4647.409154] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 4647.416443] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 4647.422787] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 4647.429236] Lustre: 250677:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 4651.102419] Lustre: server umount MGS complete [ 4651.601771] loop0: detected capacity change from 0 to 204800000 [ 4651.642838] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.878764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4651.906534] Lustre: lustre-MDT0000: Aborting client recovery [ 4652.426476] loop1: detected capacity change from 0 to 204800000 [ 4652.490652] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.533702] Lustre: 251107:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4652.540198] Lustre: 251107:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 4652.548107] Lustre: 251107:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 4652.555512] Lustre: 251107:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 4652.562117] Lustre: 251107:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 4652.568694] Lustre: 251107:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 4652.575795] Lustre: 251107:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 4652.591523] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4652.688531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4652.725788] Lustre: lustre-MDT0001: Aborting client recovery [ 4653.216478] loop2: detected capacity change from 0 to 204800000 [ 4653.298851] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4653.347842] Lustre: 251187:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4653.358303] Lustre: 251187:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 4653.366493] Lustre: 251187:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 4653.376724] Lustre: 251187:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 4653.385243] Lustre: 251187:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 4653.393533] Lustre: 251187:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 4653.400715] Lustre: 251187:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 4653.472354] Lustre: Found index 0 for lustre-OST0000, updating log [ 4653.496678] Lustre: 251207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4653.538688] Lustre: lustre-OST0000: Aborting recovery [ 4654.042000] loop3: detected capacity change from 0 to 204800000 [ 4654.091756] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.150965] Lustre: 251273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4654.157650] Lustre: 251273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 4654.165575] Lustre: 251273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 4654.172545] Lustre: 251273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 4654.178485] Lustre: 251273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 4654.184794] Lustre: 251273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 4654.190867] Lustre: 251273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 4654.218925] Lustre: 251278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4654.268796] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4654.271613] Lustre: Skipped 1 previous similar message [ 4654.288194] Lustre: lustre-OST0001: Aborting recovery [ 4659.700693] LustreError: 251051:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 4661.733098] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 4661.735344] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 4661.745243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 4661.750212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 4661.753840] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 4661.767145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 4667.028463] LustreError: 251018:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 181us: rc = -2 [ 4667.531355] LustreError: 251018:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 437us: rc = -2 [ 4667.541239] LustreError: 251018:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 37 previous similar messages [ 4682.209732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.210262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4682.220705] Lustre: Skipped 1 previous similar message [ 4682.221606] Lustre: Skipped 2 previous similar messages [ 4687.174126] Lustre: server umount lustre-MDT0000 complete [ 4687.330589] LustreError: 251131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.340861] LustreError: 251131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4687.763520] LustreError: 251001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786944228 with bad export cookie 11073711289758630172 [ 4687.772033] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4692.449220] LustreError: 251017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4692.449649] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.452266] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4692.452278] Lustre: Skipped 2 previous similar messages [ 4692.462550] LustreError: 251017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4692.471645] Lustre: Skipped 2 previous similar messages [ 4697.576812] LustreError: 251015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.577566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4697.585688] LustreError: 251015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4697.603570] Lustre: Skipped 2 previous similar messages [ 4702.178392] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4702.374618] Lustre: server umount lustre-MDT0001 complete [ 4709.143726] Lustre: server umount lustre-OST0000 complete [ 4724.192217] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4724.284397] Lustre: server umount lustre-OST0001 complete [ 4728.627835] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 01:24:27 (1786944267) [ 4743.230375] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 4747.678716] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4767.911704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.040812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4768.060801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4768.110212] Lustre: lustre-MDT0000: new disk, initializing [ 4768.168231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4768.192472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4771.231474] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4777.944457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.002336] Lustre: 255341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4778.035363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4778.042193] Lustre: Skipped 1 previous similar message [ 4778.087502] Lustre: lustre-MDT0001: new disk, initializing [ 4778.139822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4778.160068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4778.166364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4780.520121] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4786.095611] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4789.131419] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4793.083251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.249538] Lustre: lustre-OST0000: new disk, initializing [ 4793.253194] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4793.260926] Lustre: 257307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4793.335300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.257753] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4798.985207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4798.995030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4799.075906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4803.367650] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4807.039646] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4809.324131] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4817.882923] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4825.872519] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 4865.513816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.516219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4865.517335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4865.517344] Lustre: Skipped 2 previous similar messages [ 4865.520219] Lustre: Skipped 1 previous similar message [ 4870.627417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4870.633969] Lustre: Skipped 2 previous similar messages [ 4877.282038] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4877.370288] Lustre: server umount lustre-OST0000 complete [ 4880.865870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.865946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4880.866773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4880.866778] Lustre: Skipped 2 previous similar messages [ 4880.871440] Lustre: Skipped 1 previous similar message [ 4880.881187] LustreError: Skipped 1 previous similar message [ 4894.176111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4894.273429] Lustre: server umount lustre-MDT0000 complete [ 4896.080842] LustreError: 255333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786944436 with bad export cookie 11073711289758632874 [ 4896.086502] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4896.087816] LustreError: 255333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4896.245904] Lustre: server umount lustre-MDT0001 complete [ 4901.959221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.847054] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4912.422540] Lustre: server umount MGS complete [ 4927.996270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4928.826703] LustreError: 261127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4928.850664] LustreError: 261127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4928.896050] LustreError: 261140:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4929.017559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4934.115658] LustreError: 261128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.406618] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4944.354763] LustreError: 261128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4944.381284] LustreError: 261128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4945.876283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.499433] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4959.335355] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4966.253018] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4975.578635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4976.114910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4976.118830] Lustre: Skipped 1 previous similar message [ 4981.814976] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4989.348909] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5025.252296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5025.254816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.259729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5025.272945] Lustre: Skipped 1 previous similar message [ 5025.280540] Lustre: Skipped 6 previous similar messages [ 5028.498264] Lustre: server umount lustre-OST0000 complete [ 5033.441874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5033.451729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.579493] Lustre: server umount lustre-MDT0000 complete [ 5040.352890] LustreError: 261109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786944580 with bad export cookie 11073711289758634743 [ 5040.358090] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5040.358939] LustreError: 261109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5040.546331] Lustre: server umount lustre-MDT0001 complete [ 5048.736943] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5051.170379] Key type lgssc unregistered [ 5051.441521] LNet: 265095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5051.450033] LNetError: 265095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5052.518307] LNet: Removed LNI 192.168.204.124@tcp [ 5053.308322] Key type .llcrypt unregistered [ 5053.310656] Key type ._llcrypt unregistered [ 5078.168713] Key type ._llcrypt registered [ 5078.171651] Key type .llcrypt registered [ 5078.248048] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5088.455459] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5089.173171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5089.193729] alg: No test for adler32 (adler32-zlib) [ 5090.206435] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 5090.396444] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 5092.034503] Key type lgssc registered [ 5092.869775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5119.065357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5119.082505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.252904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5120.275840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5120.325376] Lustre: lustre-MDT0000: new disk, initializing [ 5120.383279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.395507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5123.811436] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5132.088237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.173660] Lustre: 269032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5132.220553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5132.223752] Lustre: Skipped 1 previous similar message [ 5132.295834] Lustre: lustre-MDT0001: new disk, initializing [ 5132.358978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5132.399794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5132.413348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5135.927309] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5143.411525] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5147.903228] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5153.409513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5153.641684] Lustre: lustre-OST0000: new disk, initializing [ 5153.647598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5153.653709] Lustre: 271000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5153.739662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5157.881783] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5162.534828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5162.541994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5162.633538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5164.880823] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5170.309102] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 5174.609782] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 5182.931973] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 5191.627351] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 5229.026580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5229.031399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.034560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.058428] Lustre: Skipped 1 previous similar message [ 5232.569512] Lustre: server umount lustre-OST0000 complete [ 5239.266767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.267873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5239.282505] Lustre: Skipped 1 previous similar message [ 5239.299819] Lustre: Skipped 2 previous similar messages [ 5241.746151] Lustre: server umount lustre-MDT0000 complete [ 5244.494770] LustreError: 269022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786944784 with bad export cookie 10689580509521901414 [ 5244.497757] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.505659] LustreError: 269022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5244.816268] Lustre: server umount lustre-MDT0001 complete [ 5251.777537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.107342] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5258.499170] Lustre: server umount MGS complete [ 5265.812635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5266.081459] LustreError: 274810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.094424] LustreError: 274810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5266.105406] LustreError: 274823:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5266.165057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5268.914616] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5271.521744] LustreError: 274811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.066093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.313283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5276.613419] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5280.702105] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5283.874257] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5287.793474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5287.997723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5291.080347] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5295.096248] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5314.529086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5314.532334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.536115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5314.542975] Lustre: Skipped 1 previous similar message [ 5318.704232] Lustre: server umount lustre-OST0000 complete [ 5320.674084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5320.677118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.683841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5320.686875] Lustre: Skipped 1 previous similar message [ 5324.771168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.774440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5324.784701] Lustre: Skipped 1 previous similar message [ 5326.630841] Lustre: server umount lustre-MDT0000 complete [ 5328.346745] LustreError: 274791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786944868 with bad export cookie 10689580509521903255 [ 5328.348525] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.353199] LustreError: 274791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5328.517750] Lustre: server umount lustre-MDT0001 complete [ 5333.708619] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5335.077885] Key type lgssc unregistered [ 5335.235476] LNet: 278756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5335.238411] LNetError: 278756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5335.247885] LNet: Removed LNI 192.168.204.124@tcp [ 5335.669321] Key type .llcrypt unregistered [ 5335.670790] Key type ._llcrypt unregistered [ 5345.232507] Key type ._llcrypt registered [ 5345.234101] Key type .llcrypt registered [ 5345.282601] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5352.421317] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5352.816560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5352.852142] alg: No test for adler32 (adler32-zlib) [ 5353.744536] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 5353.850519] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 5355.448122] Key type lgssc registered [ 5355.923317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5366.720765] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5373.519442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5373.528578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5374.625325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5374.646125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5374.694252] Lustre: lustre-MDT0000: new disk, initializing [ 5374.730233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5374.740081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5376.585825] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5381.931057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5382.064166] Lustre: lustre-OST0000: new disk, initializing [ 5382.066969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5382.069486] Lustre: Skipped 1 previous similar message [ 5382.073167] Lustre: 282796:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5382.106535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.847259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5383.851768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5383.874255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5384.843943] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5752.801166] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.809145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5757.569682] Lustre: server umount lustre-MDT0000 complete [ 5759.324835] LustreError: 282795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786945299 with bad export cookie 7747855245962582985 [ 5759.328150] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.332130] LustreError: 282795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5759.409757] Lustre: server umount lustre-OST0000 complete [ 5771.442891] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 5776.274949] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5790.850578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.944042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5790.959953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5790.998523] Lustre: lustre-MDT0000: new disk, initializing [ 5791.029450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5791.038980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5792.701981] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5797.508745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.568393] Lustre: 287054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5797.599961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5797.603202] Lustre: Skipped 1 previous similar message [ 5797.646172] Lustre: lustre-MDT0001: new disk, initializing [ 5797.694755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5797.711714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5797.718833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5799.326236] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5803.871173] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5806.166952] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5809.018993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.143791] Lustre: lustre-OST0000: new disk, initializing [ 5809.146207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5809.148876] Lustre: 289023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5809.187805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.781058] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5816.362574] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5817.354193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5817.364307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5817.414307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5818.217434] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5818.323462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5820.002239] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5820.091312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5822.437738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5822.438056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.439215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.440414] LustreError: Skipped 1 previous similar message [ 5822.445520] Lustre: Skipped 1 previous similar message [ 5826.979103] Lustre: server umount lustre-OST0000 complete [ 5832.674428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.679384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5832.681759] Lustre: Skipped 1 previous similar message [ 5832.686973] Lustre: Skipped 2 previous similar messages [ 5834.787332] Lustre: server umount lustre-MDT0000 complete [ 5836.206502] LustreError: 287048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786945376 with bad export cookie 7747855245965523762 [ 5836.209571] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.211760] LustreError: 287048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5836.346320] Lustre: server umount lustre-MDT0001 complete [ 5837.783294] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 5838.478758] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 01:42:57 (1786945377) [ 5841.293932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.531515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.269437] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5846.807864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5848.659214] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5851.689650] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5854.314475] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5857.417307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.564381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.566729] Lustre: Skipped 1 previous similar message [ 5860.177139] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5863.526970] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5866.635368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5866.724600] Lustre: lustre-OST0001: new disk, initializing [ 5866.727143] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5866.732322] Lustre: 294166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5866.836158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5866.842105] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5866.867942] LustreError: 294184:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5866.877088] LustreError: 294184:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 5867.939630] LustreError: 294191:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5867.944674] LustreError: 294191:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 5868.963428] LustreError: 294184:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5868.969399] LustreError: 294184:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 5869.414215] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5871.012169] LustreError: 294191:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5871.016957] LustreError: 294191:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 5873.971308] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5874.069470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5875.109627] LustreError: 294184:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5875.116099] LustreError: 294184:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 5876.315667] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff939451201800.ost_server_uuid 50 [ 5876.971867] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff939451201800.ost_server_uuid in FULL state after 0 sec [ 5879.251467] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff939451201800.ost_server_uuid 50 [ 5879.903684] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff939451201800.ost_server_uuid in FULL state after 0 sec [ 5881.263044] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5892.582582] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.584131] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5892.588119] Lustre: Skipped 1 previous similar message [ 5892.592810] Lustre: Skipped 1 previous similar message [ 5897.700847] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5897.705518] Lustre: Skipped 2 previous similar messages [ 5906.400266] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5906.441723] Lustre: server umount lustre-OST0001 complete [ 5907.941626] LustreError: 293151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5907.949808] LustreError: 293151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5912.033883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5912.038878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.045051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5912.047445] Lustre: Skipped 4 previous similar messages [ 5913.060121] LustreError: 294051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5913.062524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5913.069544] LustreError: 294051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5914.594821] Lustre: server umount lustre-OST0000 complete [ 5922.541716] Lustre: server umount lustre-MDT0000 complete [ 5923.298618] LustreError: 291326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.303858] LustreError: 291326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5924.316383] LustreError: 291313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786945464 with bad export cookie 7747855245965524630 [ 5924.321517] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.323797] LustreError: 291313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5924.493765] Lustre: server umount lustre-MDT0001 complete [ 5929.247967] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 5930.492305] Key type lgssc unregistered [ 5930.646975] LNet: 296845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5930.652071] LNetError: 296845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5930.662421] LNet: Removed LNI 192.168.204.124@tcp [ 5931.024922] Key type .llcrypt unregistered [ 5931.026467] Key type ._llcrypt unregistered [ 5942.379662] Key type ._llcrypt registered [ 5942.381128] Key type .llcrypt registered [ 5942.434774] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 5942.859714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5942.918721] alg: No test for adler32 (adler32-zlib) [ 5943.812827] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 5943.941100] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 5945.568270] Key type lgssc registered [ 5946.066852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5949.548204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5949.556885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.757657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.436065] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5955.585889] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5958.352522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5958.546655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5960.242708] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5964.346795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.519651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.934103] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5968.616035] LustreError: 299922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.624864] LustreError: 299922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5969.886707] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5972.689877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5972.799623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5975.458128] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 5975.914876] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5975.920381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5978.608709] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5981.771635] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5981.863510] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6068.194665] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6068.199922] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.202909] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6068.205683] Lustre: Skipped 1 previous similar message [ 6072.389104] Lustre: server umount lustre-OST0001 complete [ 6073.316910] LustreError: 301340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.321864] LustreError: 301340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6076.386509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.392343] LustreError: Skipped 1 previous similar message [ 6076.394125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.401376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.403147] Lustre: Skipped 1 previous similar message [ 6078.437290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.438032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.439419] LustreError: 299922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.439622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6081.379252] Lustre: server umount lustre-OST0000 complete [ 6083.553290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.556785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6083.564116] Lustre: Skipped 1 previous similar message [ 6083.565169] Lustre: Skipped 1 previous similar message [ 6088.680233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6088.686314] Lustre: Skipped 2 previous similar messages [ 6089.542644] Lustre: server umount lustre-MDT0000 complete [ 6093.323959] LustreError: 302024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786945633 with bad export cookie 17429774232536131185 [ 6093.331577] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6093.333348] LustreError: 302024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6093.845280] Lustre: server umount lustre-MDT0001 complete [ 6107.871285] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 6112.259547] Key type lgssc unregistered [ 6112.671691] LNet: 303185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6112.685282] LNetError: 303185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6113.771651] LNet: Removed LNI 192.168.204.124@tcp [ 6114.823517] Key type .llcrypt unregistered [ 6114.827521] Key type ._llcrypt unregistered [ 6189.041138] Key type ._llcrypt registered [ 6189.046800] Key type .llcrypt registered [ 6189.208641] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 6191.164394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6191.553471] alg: No test for adler32 (adler32-zlib) [ 6193.033525] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 6193.489787] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 6195.200115] Key type lgssc registered [ 6196.493187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6208.795718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6208.818502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.366679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.708696] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6219.756960] LustreError: 305481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6219.765947] LustreError: 305481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6224.865657] LustreError: 305480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6226.002220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6226.576470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6230.848409] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6234.053420] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6240.953201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.429862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.994879] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6251.702568] LustreError: 306969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.775799] LustreError: 306969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.789628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6254.792524] LustreError: 306969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6256.295962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6256.522464] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.761465] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 6262.676866] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6272.281503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6279.226970] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6292.465996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.475465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6292.500703] Lustre: Skipped 2 previous similar messages [ 6292.527646] Lustre: Skipped 3 previous similar messages [ 6297.580671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.598165] Lustre: Skipped 3 previous similar messages [ 6298.836286] Lustre: server umount lustre-MDT0000 complete [ 6302.698821] LustreError: 305480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.723035] LustreError: 305480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6302.859222] LustreError: 305460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786945843 with bad export cookie 2496144883860443030 [ 6302.868116] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6302.872105] LustreError: 305460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6303.285129] Lustre: server umount lustre-MDT0001 complete [ 6317.304562] Lustre: server umount lustre-OST0000 complete [ 6331.075807] Lustre: server umount lustre-OST0001 complete [ 6347.478769] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 6359.478861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.058764] LustreError: 310747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.164576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.529686] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6373.934753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.259655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6377.861550] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6380.757244] Lustre: 311887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6387.289971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6393.273447] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6398.883651] LustreError: 312241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.906432] LustreError: 312241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6403.448705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6403.689360] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6403.692626] Lustre: Skipped 1 previous similar message [ 6405.782970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 6405.785139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6410.648385] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6420.813384] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6426.488418] Lustre: 313762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6435.842604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6435.862316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.875693] Lustre: Skipped 1 previous similar message [ 6435.896929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.455701] Lustre: server umount lustre-MDT0000 complete [ 6441.958951] LustreError: 310744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.979365] LustreError: 310744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6443.464103] LustreError: 310729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786945983 with bad export cookie 2496144883860444731 [ 6443.482565] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6443.807386] Lustre: server umount lustre-MDT0001 complete [ 6458.585765] Lustre: server umount lustre-OST0000 complete [ 6463.439384] Lustre: 304740:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786945987/real 1786945987] req@ffff99808440bb80 x1873748409043456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786946003 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6474.150032] Lustre: server umount lustre-OST0001 complete [ 6479.442826] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 6481.270392] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 6483.616600] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 01:53:41 (1786946021) [ 6513.246858] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 6523.834531] loop0: detected capacity change from 0 to 1099511627776 [ 6524.544958] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6524.564074] *** VALIDATE xfs *** [ 6524.600078] XFS (loop0): Mounting V5 Filesystem [ 6524.694031] XFS (loop0): Ending clean mount [ 6526.137323] loop1: detected capacity change from 0 to 18691697672192 [ 6661.129481] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 6671.842668] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 6705.538938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.724137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6705.754337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6705.824716] Lustre: lustre-MDT0000: new disk, initializing [ 6705.886129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6705.901577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6709.488757] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6718.590829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.708766] Lustre: 318906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6718.779599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6718.786853] Lustre: Skipped 1 previous similar message [ 6718.877893] Lustre: lustre-MDT0001: new disk, initializing [ 6718.992488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6719.004099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6722.994205] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6731.252616] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6736.126545] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6742.075927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6742.277444] Lustre: lustre-OST0000: new disk, initializing [ 6742.280717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6742.286726] Lustre: 320872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6744.376919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6744.384109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6744.487261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6747.919907] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6756.076993] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6760.364121] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6760.582972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6764.948979] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6765.166857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6770.153089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.154081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6770.161901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6770.162212] Lustre: Skipped 4 previous similar messages [ 6770.169276] Lustre: Skipped 4 previous similar messages [ 6770.200948] LustreError: Skipped 1 previous similar message [ 6773.155348] Lustre: server umount lustre-OST0000 complete [ 6780.387357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.390654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6780.399185] Lustre: Skipped 1 previous similar message [ 6780.399498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6780.416299] Lustre: Skipped 2 previous similar messages [ 6782.749478] Lustre: server umount lustre-MDT0000 complete [ 6785.517465] LustreError: 318913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6785.531770] LustreError: 318913:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6785.823933] LustreError: 318897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786946326 with bad export cookie 2496144883860446285 [ 6785.826876] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6785.842434] LustreError: 318897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6786.208920] Lustre: server umount lustre-MDT0001 complete [ 6793.312534] XFS (loop0): Unmounting Filesystem [ 6797.491263] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 01:58:55 (1786946335) [ 6803.373758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.819040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6803.825989] Lustre: Skipped 2 previous similar messages [ 6806.980683] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6814.270670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.943877] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6825.183340] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6830.234671] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6836.123338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6841.520304] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6846.967720] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6859.242029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6859.243855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.252489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6859.271145] Lustre: Skipped 1 previous similar message [ 6859.280538] Lustre: Skipped 1 previous similar message [ 6865.212323] Lustre: server umount lustre-OST0000 complete [ 6869.475786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.714823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6879.730895] Lustre: Skipped 7 previous similar messages [ 6882.272143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6882.400718] Lustre: server umount lustre-MDT0000 complete [ 6884.863902] LustreError: 323412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786946425 with bad export cookie 2496144883860447167 [ 6884.873437] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6885.086774] Lustre: server umount lustre-MDT0001 complete [ 6892.521771] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 6894.574428] Key type lgssc unregistered [ 6894.817775] LNet: 327063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6894.822264] LNetError: 327063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6894.838327] LNet: Removed LNI 192.168.204.124@tcp [ 6895.515281] Key type .llcrypt unregistered [ 6895.517623] Key type ._llcrypt unregistered [ 6914.089245] Key type ._llcrypt registered [ 6914.091224] Key type .llcrypt registered [ 6914.158555] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 6914.794609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6914.828976] alg: No test for adler32 (adler32-zlib) [ 6915.756867] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 6915.933074] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 6917.600241] Key type lgssc registered [ 6918.263675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6923.618733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6923.631860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.935985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.343402] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6932.067268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.304963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6934.671528] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6939.339879] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6943.015443] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6947.509942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.754068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.592782] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6956.545693] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6962.148941] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6962.160143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6962.164506] Lustre: Skipped 1 previous similar message [ 6963.374430] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.24@tcp (stopping) [ 6967.267500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6967.272379] Lustre: Skipped 1 previous similar message [ 6967.846304] Lustre: server umount lustre-MDT0001 complete [ 6968.477754] LustreError: 328614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.492184] LustreError: 328614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6972.387597] LustreError: 329347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.410222] LustreError: 329347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6973.592713] LustreError: 330458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.004777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6975.110932] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6975.118172] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6975.152136] Lustre: 328608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6975.369834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6978.722667] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 6980.591096] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6982.644429] LustreError: 332218:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6982.651560] LustreError: 332218:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6982.656839] LustreError: 328611:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6982.694329] LustreError: 332221:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.124@tcp: cfg command failed: rc = -17 [ 6982.700609] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6982.700609] [ 7282.658925] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7282.661383] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.666492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7282.676949] Lustre: Skipped 2 previous similar messages [ 7282.685732] Lustre: Skipped 1 previous similar message [ 7287.676323] Lustre: server umount lustre-MDT0001 complete [ 7287.779848] LustreError: 328619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.788740] LustreError: 328619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7292.892635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7292.899239] LustreError: 328616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.914553] LustreError: 328616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7292.971456] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7292.974591] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7292.995112] Lustre: 328606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7293.167319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7295.200215] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7298.537545] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7300.592054] LustreError: 343303:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7300.598772] LustreError: 343303:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7300.604540] LustreError: 328611:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7300.609314] LustreError: 328611:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7300.641096] LustreError: 343306:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [ 7300.646194] LustreError: 343306:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [ 7300.652108] LustreError: 343306:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.124@tcp: cfg command failed: rc = -17 [ 7300.661430] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.124@tcp [ 7300.661430] [ 7600.612165] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.618942] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7600.629697] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7600.629708] Lustre: Skipped 1 previous similar message [ 7600.639974] Lustre: Skipped 1 previous similar message [ 7604.781571] Lustre: server umount lustre-MDT0001 complete [ 7605.731512] LustreError: 328619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.768511] LustreError: 328619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7617.077768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.170204] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7617.175413] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7617.222854] Lustre: 328608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7617.561265] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7622.663139] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7622.685251] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7623.193771] LustreError: 351876:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7623.211316] LustreError: 351876:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7623.223824] LustreError: 328611:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7623.231370] LustreError: 328611:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7623.340053] LustreError: 351885:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7623.360917] LustreError: 351885:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.124@tcp: cfg command failed: rc = -114 [ 7623.369975] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7623.369975] [ 7950.306165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7950.327195] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7950.329906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7950.329916] Lustre: Skipped 2 previous similar messages [ 7950.384270] Lustre: Skipped 2 previous similar messages [ 7956.698725] Lustre: server umount lustre-MDT0000 complete [ 7960.558409] LustreError: 328619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.608623] LustreError: 328619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7962.405074] LustreError: 331815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947502 with bad export cookie 12666578541007992027 [ 7962.407929] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7962.443960] LustreError: 331815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7965.673919] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7967.746620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7967.769973] Lustre: Skipped 6 previous similar messages [ 7969.120524] Lustre: server umount lustre-MDT0001 complete [ 7974.794176] Lustre: server umount lustre-OST0000 complete [ 7988.841713] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 8005.219716] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 8055.320341] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 02:19:52 (1786947592) [ 8064.345797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8064.674292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8064.722249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8064.908405] Lustre: lustre-MDT0000: new disk, initializing [ 8065.043657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8065.079424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8071.015598] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8083.098224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8083.297984] Lustre: 361922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8083.381488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8083.385927] Lustre: Skipped 1 previous similar message [ 8083.492920] Lustre: lustre-MDT0001: new disk, initializing [ 8083.614647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8083.665122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8083.692288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8089.790894] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8102.238857] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8110.068102] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8119.114573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8119.446149] Lustre: lustre-OST0000: new disk, initializing [ 8119.458187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8119.470873] Lustre: 363891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8119.574996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8127.674551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8127.690158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8127.937754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8128.029429] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8139.967125] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8149.984461] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8149.997613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.016521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8156.585268] Lustre: server umount lustre-MDT0000 complete [ 8158.691429] LustreError: 361933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8158.716060] LustreError: 361933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8161.567903] LustreError: 361913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947701 with bad export cookie 12666578541007993980 [ 8161.575712] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8161.576871] LustreError: 361913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8161.596078] LustreError: 362728:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 8162.207151] Lustre: server umount lustre-MDT0001 complete [ 8177.574311] Lustre: server umount lustre-OST0000 complete [ 8213.371205] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 8225.802827] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 8266.538827] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8124 sec ========== 02:23:24 (1786947804) [ 8269.040671] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:23:26 (1786947806) === [ 8273.653426] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:23:31 (1786947811) === [ 8325.042664] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 8328.646703] Key type lgssc unregistered [ 8328.961720] LNet: 370264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8328.966107] LNetError: 370264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8328.982645] LNet: Removed LNI 192.168.204.124@tcp [ 8329.936920] Key type .llcrypt unregistered [ 8329.941624] Key type ._llcrypt unregistered