[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 398820549 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002181] x2apic enabled [ 0.002945] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005719] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006009] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008075] LSM: Security Framework initializing [ 0.009025] Yama: becoming mindful. [ 0.010018] SELinux: Initializing. [ 0.010711] *** VALIDATE selinux *** [ 0.017621] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021387] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023015] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024060] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025055] *** VALIDATE tmpfs *** [ 0.026349] *** VALIDATE proc *** [ 0.027165] *** VALIDATE cgroup *** [ 0.027841] *** VALIDATE cgroup2 *** [ 0.028169] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029102] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031016] Spectre V2 : User space: Vulnerable [ 0.032003] Speculative Store Bypass: Vulnerable [ 0.034965] debug: unmapping init [mem 0xffffffff98e59000-0xffffffff98e60fff] [ 0.036077] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037430] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038009] ... version: 2 [ 0.038756] ... bit width: 48 [ 0.039005] ... generic registers: 4 [ 0.039684] ... value mask: 0000ffffffffffff [ 0.040005] ... max period: 00007fffffffffff [ 0.041005] ... fixed-purpose events: 3 [ 0.041818] ... event mask: 000000070000000f [ 0.042132] rcu: Hierarchical SRCU implementation. [ 0.044052] smp: Bringing up secondary CPUs ... [ 0.045362] x86: Booting SMP configuration: [ 0.046014] .... node #0, CPUs: #1 #2 #3 [ 0.048776] smp: Brought up 1 node, 4 CPUs [ 0.049793] smpboot: Max logical packages: 1 [ 0.050006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.232444] node 0 deferred pages initialised in 181ms [ 0.236016] devtmpfs: initialized [ 0.237152] x86/mm: Memory block size: 128MB [ 0.240390] gcov: version magic: 0x41383552 [ 0.243216] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.244054] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.245000] pinctrl core: initialized pinctrl subsystem [ 0.245000] [ 0.246005] ************************************************************* [ 0.248008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249005] ** ** [ 0.250006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.252011] ** ** [ 0.253005] ** This means that this kernel is built to expose internal ** [ 0.254006] ** IOMMU data structures, which may compromise security on ** [ 0.256006] ** your system. ** [ 0.257005] ** ** [ 0.258006] ** If you see this message and you are not debugging the ** [ 0.259006] ** kernel, report this immediately to your vendor! ** [ 0.261006] ** ** [ 0.262005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264006] ************************************************************* [ 0.265459] NET: Registered protocol family 16 [ 0.266298] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.268026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.269026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.271288] cpuidle: using governor menu [ 0.272346] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273206] PCI: Using configuration type 1 for base access [ 0.274087] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.280104] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.282143] cryptd: max_cpu_qlen set to 1000 [ 0.283213] ACPI: Added _OSI(Module Device) [ 0.284070] ACPI: Added _OSI(Processor Device) [ 0.286012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.288008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.293755] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.299529] ACPI: Interpreter enabled [ 0.301044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.302009] ACPI: Using IOAPIC for interrupt routing [ 0.304054] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.307394] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.317811] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.319020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.321008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.323041] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.325889] acpiphp: Slot [2] registered [ 0.327053] acpiphp: Slot [5] registered [ 0.327897] acpiphp: Slot [6] registered [ 0.328049] acpiphp: Slot [7] registered [ 0.328849] acpiphp: Slot [8] registered [ 0.330049] acpiphp: Slot [9] registered [ 0.330914] acpiphp: Slot [10] registered [ 0.332057] acpiphp: Slot [3] registered [ 0.332866] acpiphp: Slot [4] registered [ 0.333045] acpiphp: Slot [11] registered [ 0.333866] acpiphp: Slot [12] registered [ 0.335053] acpiphp: Slot [13] registered [ 0.335899] acpiphp: Slot [14] registered [ 0.336042] acpiphp: Slot [15] registered [ 0.336908] acpiphp: Slot [16] registered [ 0.338044] acpiphp: Slot [17] registered [ 0.339046] acpiphp: Slot [18] registered [ 0.340052] acpiphp: Slot [19] registered [ 0.340892] acpiphp: Slot [20] registered [ 0.342055] acpiphp: Slot [21] registered [ 0.343016] acpiphp: Slot [22] registered [ 0.344046] acpiphp: Slot [23] registered [ 0.346072] acpiphp: Slot [24] registered [ 0.347076] acpiphp: Slot [25] registered [ 0.349057] acpiphp: Slot [26] registered [ 0.351072] acpiphp: Slot [27] registered [ 0.353079] acpiphp: Slot [28] registered [ 0.355071] acpiphp: Slot [29] registered [ 0.358054] acpiphp: Slot [30] registered [ 0.359054] acpiphp: Slot [31] registered [ 0.360042] PCI host bridge to bus 0000:00 [ 0.362013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.364011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.366010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.368010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.370012] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.373015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.374129] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.376686] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.379745] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.387012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.390576] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.393010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.396009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.399010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.401368] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.403597] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.406029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.408502] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.412007] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.423010] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.427008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.431659] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.438009] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.442018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456012] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.465965] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.490016] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.496984] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.501017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521013] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.530932] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.538015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543009] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554012] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.563818] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.570010] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574008] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.586012] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.593093] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.602012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.606014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.618013] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.626292] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627236] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.629297] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.631236] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.633111] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.636075] iommu: Default domain type: Passthrough [ 0.637227] SCSI subsystem initialized [ 0.638098] ACPI: bus type USB registered [ 0.638955] usbcore: registered new interface driver usbfs [ 0.640032] usbcore: registered new interface driver hub [ 0.641034] usbcore: registered new device driver usb [ 0.642036] pps_core: LinuxPPS API ver. 1 registered [ 0.642947] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.645016] PTP clock support registered [ 0.646141] EDAC MC: Ver: 3.0.0 [ 0.648144] PCI: Using ACPI for IRQ routing [ 0.650924] NetLabel: Initializing [ 0.652008] NetLabel: domain hash size = 128 [ 0.654009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.657061] NetLabel: unlabeled traffic allowed by default [ 0.659096] vgaarb: loaded [ 0.661313] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.663013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.669332] clocksource: Switched to clocksource kvm-clock [ 0.796574] VFS: Disk quotas dquot_6.6.0 [ 0.798141] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.800702] *** VALIDATE ramfs *** [ 0.801972] *** VALIDATE hugetlbfs *** [ 0.804258] pnp: PnP ACPI init [ 0.806722] pnp: PnP ACPI: found 6 devices [ 0.822114] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.825413] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.827557] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.829716] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.832131] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.834552] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.837549] NET: Registered protocol family 2 [ 0.839988] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.844822] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.848303] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.854118] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.857347] TCP: Hash tables configured (established 65536 bind 65536) [ 0.860349] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.863593] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.866644] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869927] NET: Registered protocol family 1 [ 0.872492] RPC: Registered named UNIX socket transport module. [ 0.874791] RPC: Registered udp transport module. [ 0.876614] RPC: Registered tcp transport module. [ 0.878354] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.880781] NET: Registered protocol family 44 [ 0.882496] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.884105] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.885381] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.887454] PCI: CLS 0 bytes, default 64 [ 0.889373] Unpacking initramfs... [ 2.318332] debug: unmapping init [mem 0xffff8d873cc54000-0xffff8d873ffbffff] [ 2.320810] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.323370] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.325655] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.760253] Initialise system trusted keyrings [ 2.762371] Key type blacklist registered [ 2.764374] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.773660] zbud: loaded [ 2.776514] *** VALIDATE nfs *** [ 2.777941] *** VALIDATE nfs4 *** [ 2.779796] pstore: using deflate compression [ 2.783759] Platform Keyring initialized [ 2.874769] NET: Registered protocol family 38 [ 2.876227] Key type asymmetric registered [ 2.877469] Asymmetric key parser 'x509' registered [ 2.878518] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.880225] io scheduler mq-deadline registered [ 2.881088] io scheduler kyber registered [ 2.881882] io scheduler bfq registered [ 2.882930] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.884574] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.886130] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.888354] ACPI: Power Button [PWRF] [ 2.961461] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.059033] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.220272] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.311241] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.490787] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.518610] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.548922] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.558670] Non-volatile memory driver v1.3 [ 3.560097] Linux agpgart interface v0.103 [ 3.593763] virtio_blk virtio1: [vda] 132880 512-byte logical blocks (68.0 MB/64.9 MiB) [ 3.597330] vda: detected capacity change from 0 to 68034560 [ 3.611443] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.613996] vdb: detected capacity change from 0 to 1073741824 [ 3.627270] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.629803] vdc: detected capacity change from 0 to 2621440000 [ 3.641282] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.643791] vdd: detected capacity change from 0 to 2621440000 [ 3.655077] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.656625] vde: detected capacity change from 0 to 4294967296 [ 3.670459] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.672424] vdf: detected capacity change from 0 to 4294967296 [ 3.676495] libphy: Fixed MDIO Bus: probed [ 3.680394] usbcore: registered new interface driver usbserial_generic [ 3.682481] usbserial: USB Serial support registered for generic [ 3.684450] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.688414] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.690248] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.692540] mousedev: PS/2 mouse device common for all mice [ 3.695402] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.697339] rtc_cmos 00:05: RTC can wake from S4 [ 3.701425] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.702077] rtc_cmos 00:05: registered as rtc0 [ 3.705850] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.706089] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.707350] intel_pstate: CPU model not supported [ 3.712678] hid: raw HID events driver (C) Jiri Kosina [ 3.713724] usbcore: registered new interface driver usbhid [ 3.715321] usbhid: USB HID core driver [ 3.716456] drop_monitor: Initializing network drop monitor service [ 3.718796] Initializing XFRM netlink socket [ 3.720116] NET: Registered protocol family 10 [ 3.721797] Segment Routing with IPv6 [ 3.722639] NET: Registered protocol family 17 [ 3.723786] mpls_gso: MPLS GSO support [ 3.728367] RAS: Correctable Errors collector initialized. [ 3.730117] AVX version of gcm_enc/dec engaged. [ 3.731510] AES CTR mode by8 optimization enabled [ 3.803169] sched_clock: Marking stable (3803147361, 0)->(4442797638, -639650277) [ 3.806572] registered taskstats version 1 [ 3.808497] Loading compiled-in X.509 certificates [ 3.809744] zswap: loaded using pool lzo/zbud [ 3.829751] Key type big_key registered [ 3.839241] Key type encrypted registered [ 3.840544] ima: No TPM chip found, activating TPM-bypass! [ 3.842221] ima: Allocated hash algorithm: sha1 [ 3.843607] ima: No architecture policies found [ 3.845031] evm: Initialising EVM extended attributes: [ 3.846585] evm: security.selinux [ 3.847592] evm: security.ima [ 3.848467] evm: security.capability [ 3.849516] evm: HMAC attrs: 0x1 [ 3.851518] rtc_cmos 00:05: setting system clock to 2025-08-20 13:10:26 UTC (1755695426) [ 3.857213] debug: unmapping init [mem 0xffffffff99e03000-0xffffffff99ffffff] [ 3.859752] debug: unmapping init [mem 0xffffffff98b82000-0xffffffff98e58fff] [ 3.872108] Write protecting the kernel read-only data: 28672k [ 3.875624] debug: unmapping init [mem 0xffffffff97203000-0xffffffff973fffff] [ 3.878701] debug: unmapping init [mem 0xffffffff97b14000-0xffffffff97bfffff] [ 3.910826] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.915659] systemd[1]: Detected virtualization kvm. [ 3.916831] systemd[1]: Detected architecture x86-64. [ 3.918298] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.943300] systemd[1]: No hostname configured. [ 3.945052] systemd[1]: Set hostname to . [ 3.947126] random: systemd: uninitialized urandom read (16 bytes read) [ 3.948882] systemd[1]: Initializing machine ID from random generator. [ 3.984460] random: ln: uninitialized urandom read (6 bytes read) [ 4.059329] random: systemd: uninitialized urandom read (16 bytes read) [ 4.061582] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.064624] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.068146] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. [ 4.475209] random: fast init done Starting dracut pre-udev hook... [ 4.526459] device-mapper: uevent: version 1.0.3 [ 4.528429] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.161508] virtio_net virtio0 ens2: renamed from eth0 [ 5.171820] scsi host0: ata_piix Starting dracut initqueue hook... [ 5.188543] scsi host1: ata_piix [ 5.190559] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.193730] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.522573] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.353494] random: crng init done [ 10.354979] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.572795] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.602855] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.784149] SELinux: Disabled at runtime. [ 11.836373] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.842126] systemd[1]: Detected virtualization kvm. [ 11.843246] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.213184] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.216315] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.220456] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.224204] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.228047] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.234888] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.237191] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ 12.303326] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.553856] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.801717] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.857430] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.954958] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.971702] EDAC sbridge: Ver: 1.1.2 [ 14.159951] Key type dns_resolver registered [ 14.476267] NFS: Registering the id_resolver key type [ 14.477982] Key type id_resolver registered [ 14.479286] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg158-server login: [ 36.768850] libcfs: loading out-of-tree module taints kernel. [ 36.779373] Key type ._llcrypt registered [ 36.780212] Key type .llcrypt registered [ 36.820679] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 43.918900] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 44.437249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 44.443959] alg: No test for adler32 (adler32-zlib) [ 45.417545] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 45.684726] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 47.296196] Key type lgssc registered [ 47.838658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 53.896079] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 55.017161] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 58.631778] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 61.015872] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.507722] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 68.727367] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 73.602493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.643484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 73.662276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 74.780300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 74.793228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 74.835631] Lustre: lustre-MDT0000: new disk, initializing [ 74.865608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 74.875500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 76.349691] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 82.298455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.334285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 82.358248] Lustre: 6469:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 82.369916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 82.371428] Lustre: Skipped 1 previous similar message [ 82.409499] Lustre: lustre-MDT0001: new disk, initializing [ 82.434853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 82.449795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 82.455104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 84.017952] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 86.552740] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 90.761128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.809363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 90.955633] Lustre: lustre-OST0000: new disk, initializing [ 90.957875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 90.983870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.345267] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.728168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 95.731548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 95.760919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 99.139352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.166736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.205358] Lustre: lustre-OST0001: new disk, initializing [ 99.207618] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 99.229866] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.176549] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.619416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 101.623483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 101.650534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 107.675702] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 111.741415] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 118.921694] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing check_logdir /tmp/testlogs/ [ 120.332712] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing yml_node [ 122.205707] Lustre: DEBUG MARKER: Client: 2.16.57.97 [ 123.323871] Lustre: DEBUG MARKER: MDS: 2.16.57.97 [ 124.376408] Lustre: DEBUG MARKER: OSS: 2.16.57.97 [ 125.025161] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 20 09:12:26 EDT 2025 [ 131.821180] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 132.412630] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 137.697539] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.698151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.704587] Lustre: Skipped 2 previous similar messages [ 137.710853] Lustre: Skipped 3 previous similar messages [ 139.850666] LustreError: 12117:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 139.889918] Lustre: server umount lustre-MDT0000 complete [ 141.431709] LustreError: 6461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695564 with bad export cookie 3327334373340493721 [ 141.433241] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 141.434946] LustreError: 6461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 142.817346] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 142.817732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 142.825060] Lustre: Skipped 2 previous similar messages [ 142.830989] Lustre: Skipped 1 previous similar message [ 146.848963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 146.852279] Lustre: Skipped 1 previous similar message [ 147.584799] LustreError: 12318:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 147.587484] LustreError: 12318:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 147.692050] Lustre: server umount lustre-MDT0001 complete [ 149.164136] LustreError: 12520:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 149.168430] LustreError: 12520:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 149.193108] Lustre: server umount lustre-OST0000 complete [ 150.730982] Lustre: server umount lustre-OST0001 complete [ 152.880649] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 156.487884] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 160.659724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.929644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.686675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 169.236359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.193558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.226153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.347430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.363467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.419207] Lustre: lustre-MDT0000: new disk, initializing [ 172.460515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.472292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.077881] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 178.955390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.979863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 179.013677] Lustre: 14886:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.109966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 180.361531] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 183.770336] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 185.082019] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 187.780253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.822234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.959056] Lustre: lustre-OST0000: new disk, initializing [ 187.962093] Lustre: Skipped 1 previous similar message [ 187.965279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.967568] Lustre: Skipped 2 previous similar messages [ 188.016700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.020282] Lustre: Skipped 1 previous similar message [ 189.705032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 189.709382] Lustre: Skipped 1 previous similar message [ 189.712388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.761449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.229440] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 193.848515] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 195.417250] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 195.534373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 197.292252] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 197.405417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 200.162508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 200.165912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 200.168513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 200.169334] Lustre: Skipped 1 previous similar message [ 204.465663] LustreError: 17798:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 204.471741] LustreError: 17798:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 204.567155] Lustre: server umount lustre-OST0000 complete [ 209.889891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 209.897272] LustreError: Skipped 1 previous similar message [ 209.900264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 209.910335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 209.913410] Lustre: Skipped 1 previous similar message [ 212.053951] LustreError: 17999:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 212.056379] LustreError: 17999:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 212.097482] Lustre: server umount lustre-MDT0000 complete [ 214.002530] LustreError: 15663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695636 with bad export cookie 3327334373340495254 [ 214.009546] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 214.010090] LustreError: 15663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 214.226983] Lustre: server umount lustre-MDT0001 complete [ 216.702583] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:13:58 (1755695638) [ 219.240496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.468249] LustreError: 18726:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 219.477950] LustreError: 18726:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 219.508322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.092940] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 224.738084] LustreError: 18726:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 224.941115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 226.608647] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 228.753380] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 230.059948] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 232.820615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 234.943177] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 236.767707] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 249.444544] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 255.459081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.462256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 255.466350] Lustre: Skipped 2 previous similar messages [ 255.469374] Lustre: Skipped 2 previous similar messages [ 258.766167] LustreError: 21031:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 258.769882] LustreError: 21031:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 258.816188] Lustre: server umount lustre-MDT0000 complete [ 260.582977] LustreError: 20662:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.590285] LustreError: 20662:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 260.596991] LustreError: 18706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695683 with bad export cookie 3327334373340496136 [ 260.605471] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 260.801466] Lustre: server umount lustre-MDT0001 complete [ 272.867830] Lustre: server umount lustre-OST0000 complete [ 280.369564] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 284.369708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 284.529474] LustreError: 22753:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.544273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 284.546047] Lustre: Skipped 2 previous similar messages [ 285.810832] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 288.751323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 290.051451] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 291.299636] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 294.093394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 296.383415] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 300.509172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.556185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 300.665490] Lustre: lustre-OST0001: new disk, initializing [ 300.669162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 302.678100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 302.681366] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 302.728040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 303.205162] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 308.310473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 310.022276] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 323.556263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.558255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.561965] Lustre: Skipped 4 previous similar messages [ 323.564339] Lustre: Skipped 4 previous similar messages [ 326.095049] LustreError: 26211:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 326.099142] LustreError: 26211:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 326.138397] Lustre: server umount lustre-MDT0000 complete [ 327.591569] LustreError: 22732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695750 with bad export cookie 3327334373340496983 [ 327.598669] LustreError: 22732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 327.599069] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.788442] Lustre: server umount lustre-MDT0001 complete [ 339.905213] Lustre: server umount lustre-OST0000 complete [ 352.030058] Lustre: server umount lustre-OST0001 complete [ 358.847347] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 363.033120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.296720] LustreError: 27985:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.339315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 363.343369] Lustre: Skipped 3 previous similar messages [ 365.073078] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 369.041524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 371.140654] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 372.596523] Lustre: 29090:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 375.964980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 378.932677] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 379.248132] LustreError: 29821:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.266225] LustreError: 29821:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 383.044939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 385.614895] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 394.193379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 395.803332] Lustre: 30934:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 402.918567] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 402.921178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 402.925473] Lustre: Skipped 2 previous similar messages [ 402.928457] Lustre: Skipped 2 previous similar messages [ 408.717637] LustreError: 31135:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 408.719944] LustreError: 31135:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 408.754607] Lustre: server umount lustre-MDT0000 complete [ 410.207352] LustreError: 27966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695832 with bad export cookie 3327334373340498278 [ 410.210549] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.211573] LustreError: 27966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 410.356900] Lustre: server umount lustre-MDT0001 complete [ 422.413894] Lustre: server umount lustre-OST0000 complete [ 434.388638] Lustre: server umount lustre-OST0001 complete [ 437.256759] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:17:39 (1755695859) [ 448.162397] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 451.990688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.165769] LustreError: 33709:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.692447] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 457.311063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 459.031318] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 460.358478] Lustre: 34817:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 463.351992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 465.600330] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 469.364626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 471.909255] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 480.405719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 482.187493] Lustre: 36656:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 492.005389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.007074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.009547] Lustre: Skipped 4 previous similar messages [ 492.011860] Lustre: Skipped 6 previous similar messages [ 494.925751] LustreError: 36809:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 494.929618] LustreError: 36809:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 494.969186] Lustre: server umount lustre-MDT0000 complete [ 495.010300] LustreError: 33709:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.015889] LustreError: 33709:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 496.575709] LustreError: 34441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695919 with bad export cookie 3327334373340499384 [ 496.581196] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.582405] LustreError: 34441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 496.737797] Lustre: server umount lustre-MDT0001 complete [ 507.393948] Lustre: server umount lustre-OST0000 complete [ 518.769891] Lustre: server umount lustre-OST0001 complete [ 521.817818] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:19:03 (1755695943) [ 527.223799] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 530.990968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.247137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.251611] Lustre: Skipped 7 previous similar messages [ 532.801489] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 536.178296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 537.542775] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 538.549173] Lustre: 39879:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 541.223964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 543.530769] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 546.874611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 549.196053] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 563.481654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 570.168789] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 570.170588] Lustre: Skipped 1 previous similar message [ 571.365979] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.371403] Lustre: Skipped 3 previous similar messages [ 577.515409] Lustre: server umount lustre-MDT0000 complete [ 578.745185] LustreError: 38753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696001 with bad export cookie 3327334373340500490 [ 578.750524] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.206449] LustreError: 39488:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.215785] LustreError: 39488:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 593.376112] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 593.488776] Lustre: server umount lustre-MDT0001 complete [ 596.840359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.129066] LustreError: 3618:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8d87b8fd9500 x1840980131370240/t0(0) o250->MGC192.168.201.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 605.739133] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 609.002412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 610.868000] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 613.229054] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 618.479043] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 622.330478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 623.778441] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 624.405186] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 628.711664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.713478] Lustre: Skipped 20 previous similar messages [ 632.277405] LustreError: 44126:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 632.281945] LustreError: 44126:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 632.320679] Lustre: server umount lustre-MDT0000 complete [ 633.861335] LustreError: 38753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696056 with bad export cookie 3327334373340501666 [ 633.864864] LustreError: 38753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 633.865709] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.016998] Lustre: server umount lustre-MDT0001 complete [ 646.153206] Lustre: server umount lustre-OST0000 complete [ 658.155972] Lustre: server umount lustre-OST0001 complete [ 661.978356] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:21:23 (1755696083) [ 668.131392] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 672.594961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.156916] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 677.209384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 678.773523] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 680.088674] Lustre: 47204:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 680.095619] Lustre: 47204:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 682.729181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 685.203077] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 688.807650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 691.024100] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 705.676963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 712.616600] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 718.305557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.311837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.320807] Lustre: Skipped 6 previous similar messages [ 723.575353] Lustre: server umount lustre-MDT0000 complete [ 723.940948] LustreError: 46096:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 723.950722] LustreError: 46096:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 18 previous similar messages [ 725.295975] LustreError: 46077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696147 with bad export cookie 3327334373340502450 [ 725.303903] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.304671] LustreError: 46077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 725.482308] Lustre: server umount lustre-MDT0001 complete [ 737.294537] Lustre: server umount lustre-OST0000 complete [ 749.065261] Lustre: server umount lustre-OST0001 complete [ 753.544155] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:22:55 (1755696175) [ 754.672713] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 755.432110] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:22:57 (1755696177) [ 760.005708] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 763.769315] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 768.207135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 771.240567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 774.440489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 774.477796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.606219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 774.645792] Lustre: lustre-MDT0000: new disk, initializing [ 774.673147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 776.358210] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 780.478508] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 783.834053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 783.877771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 785.624675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 785.644480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 786.590921] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 790.479830] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 791.532661] Lustre: DEBUG MARKER: create test files [ 806.371805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 810.384709] Lustre: server umount lustre-OST0000 complete [ 812.102800] Lustre: server umount lustre-MDT0000 complete [ 816.168709] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 817.642178] Key type lgssc unregistered [ 817.786636] LNet: 54590:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.790452] LNetError: 54590:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.798444] LNet: Removed LNI 192.168.201.158@tcp [ 818.299134] Key type .llcrypt unregistered [ 818.301383] Key type ._llcrypt unregistered [ 829.923815] Key type ._llcrypt registered [ 829.924717] Key type .llcrypt registered [ 829.956986] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 830.233918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 830.267026] alg: No test for adler32 (adler32-zlib) [ 831.115116] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 831.197226] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 832.776304] Key type lgssc registered [ 833.249479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 836.145381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 836.152668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.355845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 839.063885] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 841.178137] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 844.377667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 844.540879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.572407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 846.616692] Lustre: 57160:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 846.619972] Lustre: 57160:0:(osd_handler.c:1962:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 846.622425] Lustre: 57160:0:(osd_handler.c:1969:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 846.624832] Lustre: 57160:0:(osd_handler.c:1979:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 846.627320] Lustre: 57160:0:(osd_handler.c:1986:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 846.629717] Lustre: 57160:0:(osd_handler.c:1993:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 846.866564] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 849.110162] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 852.391806] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 854.498277] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 857.059409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 857.060136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.072408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 862.179070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 862.965143] LustreError: 57896:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 863.058809] Lustre: server umount lustre-OST0000 complete [ 864.952749] LustreError: 58098:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 864.954693] LustreError: 58098:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 865.066581] Lustre: server umount lustre-MDT0000 complete [ 869.697974] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 871.290851] Key type lgssc unregistered [ 871.442601] LNet: 58626:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.445674] LNetError: 58626:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.454494] LNet: Removed LNI 192.168.201.158@tcp [ 871.856149] Key type .llcrypt unregistered [ 871.857860] Key type ._llcrypt unregistered [ 883.900202] Key type ._llcrypt registered [ 883.901416] Key type .llcrypt registered [ 883.939808] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 884.365130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 884.381351] alg: No test for adler32 (adler32-zlib) [ 885.227592] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 885.301184] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 886.872130] Key type lgssc registered [ 887.228233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 890.448295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 890.457968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.605908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.740129] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 894.807623] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.217054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 897.332964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.366676] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 901.415845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 901.449275] Lustre: 61294:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 901.456177] Lustre: 61294:0:(osd_handler.c:1962:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 901.460230] Lustre: 61294:0:(osd_handler.c:1969:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 901.465141] Lustre: 61294:0:(osd_handler.c:1979:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 901.468282] Lustre: 61294:0:(osd_handler.c:1986:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 901.470445] Lustre: 61294:0:(osd_handler.c:1993:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 901.483230] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 904.831409] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 942.564334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.564735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 942.572804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.053185] LustreError: 62008:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 944.157581] Lustre: server umount lustre-OST0000 complete [ 945.974151] LustreError: 62210:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 945.977375] LustreError: 62210:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 946.093533] Lustre: server umount lustre-MDT0000 complete [ 950.558486] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 951.962732] Key type lgssc unregistered [ 952.114478] LNet: 62737:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.118418] LNetError: 62737:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.131540] LNet: Removed LNI 192.168.201.158@tcp [ 952.550370] Key type .llcrypt unregistered [ 952.551582] Key type ._llcrypt unregistered [ 959.436867] Key type ._llcrypt registered [ 959.438018] Key type .llcrypt registered [ 959.474766] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 966.035678] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 966.450642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.457814] alg: No test for adler32 (adler32-zlib) [ 967.304585] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 967.382208] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 968.960180] Key type lgssc registered [ 969.376791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 972.425632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 974.821039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 976.929194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 979.130267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 981.421259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 981.439867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 981.445982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.524680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 982.537390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 982.577144] Lustre: lustre-MDT0000: new disk, initializing [ 982.601600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.609839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 983.733657] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 988.133263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 988.154063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 988.184697] Lustre: 65826:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 988.205206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 988.207285] Lustre: Skipped 1 previous similar message [ 988.243948] Lustre: lustre-MDT0001: new disk, initializing [ 988.275413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 988.286428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 988.290853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 989.750657] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 992.790576] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 994.019658] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 996.485043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 996.509952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 996.610068] Lustre: lustre-OST0000: new disk, initializing [ 996.612118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 996.640490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.474501] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1001.685527] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1003.089034] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1004.035536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1004.041458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1004.103698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1004.238921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1005.855942] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1005.926701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1009.127093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.127849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.131265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.132516] LustreError: Skipped 1 previous similar message [ 1009.136981] Lustre: Skipped 1 previous similar message [ 1012.654179] LustreError: 68739:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1012.721597] Lustre: server umount lustre-OST0000 complete [ 1014.242935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1014.243431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.244444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.252712] Lustre: Skipped 2 previous similar messages [ 1019.362194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.365383] Lustre: Skipped 2 previous similar messages [ 1020.158504] LustreError: 68940:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1020.161793] LustreError: 68940:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1020.196451] Lustre: server umount lustre-MDT0000 complete [ 1021.497141] LustreError: 65817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696444 with bad export cookie 14220983977110793979 [ 1021.499975] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1021.500457] LustreError: 65817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1021.568210] LustreError: 69144:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1021.570198] LustreError: 69144:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1021.647270] Lustre: server umount lustre-MDT0001 complete [ 1024.784533] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:27:26 (1755696446) [ 1029.270717] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1034.265066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1044.208472] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 1048.274911] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1052.051515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1055.056263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1057.844332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1060.737836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1064.064259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1064.093664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.207290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1064.226571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1064.272209] Lustre: lustre-MDT0000: new disk, initializing [ 1064.300825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.307512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1065.980852] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1071.062900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1071.085565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1071.122767] Lustre: 73356:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1071.151323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1071.153833] Lustre: Skipped 1 previous similar message [ 1071.191642] Lustre: lustre-MDT0001: new disk, initializing [ 1071.224367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.237986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1071.241040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1072.544912] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1075.730464] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1077.004517] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1079.591115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1079.617223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1079.720123] Lustre: lustre-OST0000: new disk, initializing [ 1079.721948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1079.753020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.988438] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1084.408694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1084.411711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1084.443210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1085.494378] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1087.075136] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1087.161101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1088.535659] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1088.626925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1089.506423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1089.507434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.508727] LustreError: Skipped 1 previous similar message [ 1089.512745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1089.516108] Lustre: Skipped 1 previous similar message [ 1091.310927] LustreError: 76262:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1091.313663] LustreError: 76262:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1091.381271] Lustre: server umount lustre-OST0000 complete [ 1094.627051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1094.628054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1094.631776] Lustre: Skipped 2 previous similar messages [ 1094.635102] Lustre: Skipped 1 previous similar message [ 1098.893714] LustreError: 76462:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1098.896576] LustreError: 76462:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1098.935329] Lustre: server umount lustre-MDT0000 complete [ 1100.438061] LustreError: 74131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696523 with bad export cookie 14220983977110794847 [ 1100.441031] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.441266] LustreError: 74131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1100.589578] Lustre: server umount lustre-MDT0001 complete [ 1103.751759] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:28:45 (1755696525) [ 1106.533987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.709950] LustreError: 77333:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.716539] LustreError: 77333:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1106.738684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.332858] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1111.190060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1112.899400] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1114.981214] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1116.180489] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1119.095560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1121.535436] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1124.066528] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1125.784298] Lustre: 79654:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting 9a2e405a-12c8-4ef2-80c8-19da793c879c at adminstrative request [ 1126.373367] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1126.384492] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1136.614468] Lustre: MGS: Client 9a2e405a-12c8-4ef2-80c8-19da793c879c (at 0@lo) reconnecting [ 1136.617430] Lustre: Evicted from MGS (at 192.168.201.158@tcp) after server handle changed from 0xc55b1594c835f5d1 to 0xc55b1594c835f85c [ 1136.623051] Lustre: MGC192.168.201.158@tcp: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 1158.423359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1158.453251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1158.507228] Lustre: lustre-OST0001: new disk, initializing [ 1158.509074] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1158.538881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1158.541380] Lustre: Skipped 2 previous similar messages [ 1160.425070] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1163.559109] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1164.557113] LustreError: 80772:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1164.559115] LustreError: 80772:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1164.574629] Lustre: server umount lustre-OST0001 complete [ 1164.807448] LustreError: 78918:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.330348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1167.330382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.332403] LustreError: Skipped 1 previous similar message [ 1167.333157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.341651] Lustre: Skipped 1 previous similar message [ 1169.898595] LustreError: 78918:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.904160] LustreError: 78918:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1172.293094] Lustre: server umount lustre-OST0000 complete [ 1179.753281] Lustre: server umount lustre-MDT0000 complete [ 1180.130542] LustreError: 78049:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.134646] LustreError: 78049:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1181.037076] LustreError: 77357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696603 with bad export cookie 14220983977110796380 [ 1181.039664] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1181.041079] LustreError: 77357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1181.106392] LustreError: 81387:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 1181.108311] LustreError: 81387:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 1181.197960] Lustre: server umount lustre-MDT0001 complete [ 1183.990973] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:30:05 (1755696605) [ 1184.512418] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1185.095493] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:30:06 (1755696606) [ 1185.618111] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1186.411381] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:30:08 (1755696608) [ 1187.037709] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1187.705396] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:30:09 (1755696609) [ 1188.238712] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1190.136403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1190.611917] loop: module loaded [ 1190.616918] loop0: detected capacity change from 0 to 4294967296 [ 1190.635522] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1191.236517] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1192.381385] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1193.769399] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1197.062285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1198.515626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1199.934592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1201.554473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1204.137948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1204.168570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1204.184230] LustreError: 83948:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1204.189315] LustreError: 83948:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1204.194468] LustreError: 83948:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1204.201521] LustreError: 83948:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1211.612935] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 1214.440068] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1217.969836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1220.145925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1222.179776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1224.291583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1226.623782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1226.655833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.753547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1226.767169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1226.809426] Lustre: lustre-MDT0000: new disk, initializing [ 1226.833498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.841383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1228.036876] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1232.323392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1232.354249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1232.392367] Lustre: 86771:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1232.496941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1233.826211] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1236.933547] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1238.177421] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1240.648526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1240.679778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1243.194404] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1245.182264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1245.230602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1246.848442] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1248.570760] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1248.658941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1250.146462] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1250.265055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1255.394806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.396207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1255.398812] Lustre: Skipped 2 previous similar messages [ 1255.401047] LustreError: Skipped 1 previous similar message [ 1255.401454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1255.405565] Lustre: Skipped 6 previous similar messages [ 1257.072539] LustreError: 89681:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1257.075214] LustreError: 89681:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1257.160022] Lustre: server umount lustre-OST0000 complete [ 1273.312206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1273.404486] Lustre: server umount lustre-MDT0000 complete [ 1274.996837] LustreError: 86762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696697 with bad export cookie 14220983977110796968 [ 1275.002084] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.003737] LustreError: 86762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1275.150745] Lustre: server umount lustre-MDT0001 complete [ 1278.849706] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 09:31:40 (1755696700) [ 1281.780127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.961116] LustreError: 90758:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.321585] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1286.336241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1287.704874] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1289.774246] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1291.242972] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1293.647257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1295.643764] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1297.611293] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1298.995301] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1307.340375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1308.593154] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1308.659908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1310.812901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1310.838747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1310.901510] Lustre: lustre-OST0001: new disk, initializing [ 1310.902926] Lustre: Skipped 2 previous similar messages [ 1310.904502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1310.906451] Lustre: Skipped 3 previous similar messages [ 1312.916684] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1316.076912] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1317.440203] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1317.890583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1317.892804] Lustre: Skipped 2 previous similar messages [ 1317.895093] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1317.935148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1318.600519] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1320.375094] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1320.490763] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1331.226078] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1339.381654] Lustre: Failing over lustre-MDT0000 [ 1339.729404] LustreError: 95657:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1339.731388] LustreError: 95657:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1339.764461] Lustre: server umount lustre-MDT0000 complete [ 1342.565584] LustreError: 90783:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.946235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1342.951180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.962530] Lustre: Skipped 2 previous similar messages [ 1354.827616] LDISKFS-fs (dm-0): recovery complete [ 1354.828964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.864673] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1354.952287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 1354.954187] Lustre: Skipped 6 previous similar messages [ 1354.981182] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1356.302782] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1357.909935] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1357.910784] Lustre: 90783:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1360.354831] Lustre: 90758:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1360.357990] Lustre: 90758:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1 previous similar message [ 1360.366289] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 1360.384954] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1360.696118] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1361.017891] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1361.020235] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1361.336125] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1361.339912] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1361.659590] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1361.666692] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 8 previous similar messages [ 1362.299206] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1362.304123] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 3 previous similar messages [ 1362.624132] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1362.628951] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 3 previous similar messages [ 1363.880889] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1363.885452] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 6 previous similar messages [ 1364.512906] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1364.516858] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 1364.832112] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1364.834582] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 1367.961990] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1367.965293] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 12 previous similar messages [ 1368.594124] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1368.597109] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 1368.912108] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1368.914510] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 1376.132564] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1376.147785] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 25 previous similar messages [ 1376.782396] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1376.792302] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 1377.112125] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1377.115646] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 1392.201245] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1392.211127] Lustre: 96402:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 50 previous similar messages [ 1392.857225] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1392.863735] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 50 previous similar messages [ 1393.184121] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1393.187105] LustreError: 96402:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 50 previous similar messages [ 1417.216100] Lustre: 96402:0:(ldlm_lib.c:2309:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1417.218995] Lustre: 96402:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1417.222060] Lustre: 96402:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 101ae13b-5770-4d1a-955a-66c73febe7d5@192.168.201.58@tcp [ 1417.226254] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1417.229691] Lustre: 96402:0:(ldlm_lib.c:1831:abort_req_replay_queue()) @@@ aborted: req@000000001d2a069a x1840981090569600/t0(8589934778) o35->101ae13b-5770-4d1a-955a-66c73febe7d5@192.168.201.58@tcp:220/0 lens 392/0 e 0 to 0 dl 1755696850 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1417.238873] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1417.242586] Lustre: lustre-MDT0000: Denying connection for new client 101ae13b-5770-4d1a-955a-66c73febe7d5 (at 192.168.201.58@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1417.246089] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1417.260169] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1417.262845] Lustre: Skipped 2 previous similar messages [ 1417.264597] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1417.278230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 1417.278332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 1422.817696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1422.820460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.824077] Lustre: Skipped 3 previous similar messages [ 1422.825940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.827688] Lustre: Skipped 6 previous similar messages [ 1428.544737] Lustre: server umount lustre-OST0000 complete [ 1432.038071] LustreError: 92345:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.045380] LustreError: 92345:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 19 previous similar messages [ 1444.320099] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1444.425164] Lustre: server umount lustre-OST0001 complete [ 1460.192129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1460.258822] Lustre: server umount lustre-MDT0000 complete [ 1461.550290] LustreError: 90737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696884 with bad export cookie 14220983977110848152 [ 1461.552645] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.556838] LustreError: 90737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1461.700586] Lustre: server umount lustre-MDT0001 complete [ 1464.754323] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:34:46 (1755696886) [ 1467.344510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1467.349778] Lustre: *** cfs_fail_loc=197, val=0*** [ 1484.832634] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1490.976522] LustreError: 98356:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 1496.103972] Lustre: *** cfs_fail_loc=197, val=0*** [ 1496.105237] Lustre: Skipped 65 previous similar messages [ 1521.621484] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1523.515430] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1524.220590] LustreError: 99015:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1524.224182] LustreError: 99015:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1524.288793] Lustre: server umount lustre-OST0000 complete [ 1527.222970] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:35:49 (1755696949) [ 1540.918326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.090189] LustreError: 99856:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.099336] LustreError: 99856:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1542.418802] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1545.528753] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1547.627593] LustreError: 99886:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1547.631847] LustreError: 99886:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1547.635768] LustreError: 99886:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1547.820908] Lustre: server umount lustre-MDT0000 complete [ 1553.418409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1559.317228] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 1563.451379] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1567.239822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1569.642929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1571.732572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1573.973937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1576.278506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1576.300143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.379178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1576.389584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1576.425140] Lustre: lustre-MDT0000: new disk, initializing [ 1576.452083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1577.779244] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1582.686581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1582.714931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1582.750079] Lustre: 103902:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1582.912608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1584.383751] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1587.751543] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1589.247337] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1591.512205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1591.545329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1593.364904] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1596.198876] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.400525] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1597.465761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1598.488424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1599.723885] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1599.789931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1602.532143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1602.534310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.534795] LustreError: Skipped 3 previous similar messages [ 1602.541147] Lustre: Skipped 5 previous similar messages [ 1602.542063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1602.546822] Lustre: Skipped 17 previous similar messages [ 1606.627492] Lustre: server umount lustre-OST0000 complete [ 1614.368223] Lustre: server umount lustre-MDT0000 complete [ 1615.602155] LustreError: 103894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697038 with bad export cookie 14220983977110858799 [ 1615.604971] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.605496] LustreError: 103894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1615.741108] Lustre: server umount lustre-MDT0001 complete [ 1619.168385] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:37:20 (1755697040) [ 1622.453016] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1623.412082] Key type lgssc unregistered [ 1623.550362] LNet: 107945:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.553353] LNetError: 107945:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.561410] LNet: Removed LNI 192.168.201.158@tcp [ 1623.889848] Key type .llcrypt unregistered [ 1623.891479] Key type ._llcrypt unregistered [ 1630.924041] Key type ._llcrypt registered [ 1630.925608] Key type .llcrypt registered [ 1630.973938] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 1638.293059] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1638.657958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.707078] alg: No test for adler32 (adler32-zlib) [ 1639.563258] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1639.648455] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1641.232144] Key type lgssc registered [ 1641.694605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1645.679897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1648.482114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1651.597178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1654.291633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1656.617721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1659.547223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1659.569214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1659.575414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.658414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1660.670291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1660.712202] Lustre: lustre-MDT0000: new disk, initializing [ 1660.735458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.742270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1662.123602] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1666.587108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1666.616573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1666.740901] Lustre: lustre-OST0000: new disk, initializing [ 1666.745142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1666.749031] Lustre: Skipped 1 previous similar message [ 1666.777940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.238463] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1670.233145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1670.239523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1670.252668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1673.825819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1673.858060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1673.931707] Lustre: lustre-OST0001: new disk, initializing [ 1673.935405] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1673.966663] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1676.366680] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1678.382886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1678.387300] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1678.399858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1683.426334] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.427078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.429374] Lustre: Skipped 1 previous similar message [ 1683.432471] Lustre: Skipped 1 previous similar message [ 1687.235695] LustreError: 113467:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1687.322292] Lustre: server umount lustre-MDT0000 complete [ 1690.047177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1692.143272] LustreError: 111240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697114 with bad export cookie 18389437486649212235 [ 1692.146696] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.149152] LustreError: 111240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1692.171246] LustreError: 114060:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1692.174875] LustreError: 114060:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1692.194326] Lustre: server umount lustre-OST0000 complete [ 1693.805042] LustreError: 114260:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1693.807790] LustreError: 114260:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1693.897644] Lustre: server umount lustre-OST0001 complete [ 1697.906147] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 09:38:39 (1755697119) [ 1702.422620] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1706.755550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1707.227029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1710.169964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1710.660944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1714.218653] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:38:55 (1755697135) [ 1720.518240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1721.515839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1722.485781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1723.436751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1723.922962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1724.875753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1732.085389] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 1735.768194] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1739.384933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1741.822378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1744.092424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1746.282912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1748.535955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1748.560096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.633296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1748.644967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1748.675406] Lustre: lustre-MDT0000: new disk, initializing [ 1748.696848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.702179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1749.953621] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1754.015542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1754.040963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1754.069658] Lustre: 119557:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1754.091427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1754.093032] Lustre: Skipped 1 previous similar message [ 1754.123844] Lustre: lustre-MDT0001: new disk, initializing [ 1754.154681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1754.166115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1754.170641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1755.411972] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1758.515841] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1759.853548] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1762.343654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1762.366649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1762.461203] Lustre: lustre-OST0000: new disk, initializing [ 1762.463671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1762.489466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.940166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1762.944141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1762.975029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1764.340187] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1767.492448] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1769.012267] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1769.103591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1770.518429] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1770.594499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1773.538712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1773.538954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.541919] LustreError: Skipped 1 previous similar message [ 1773.547430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1773.551678] Lustre: Skipped 1 previous similar message [ 1777.390221] LustreError: 122466:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1777.393122] LustreError: 122466:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1777.459752] Lustre: server umount lustre-OST0000 complete [ 1779.680854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1779.683543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.688813] Lustre: Skipped 1 previous similar message [ 1779.691146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1783.777132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1783.777647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.897103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.900030] Lustre: Skipped 2 previous similar messages [ 1792.992139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1793.025722] LustreError: 122667:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1793.028077] LustreError: 122667:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1793.066895] Lustre: server umount lustre-MDT0000 complete [ 1794.306210] LustreError: 119548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697216 with bad export cookie 18389437486649213453 [ 1794.308807] LustreError: 119548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1794.309523] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1794.454461] Lustre: server umount lustre-MDT0001 complete [ 1797.345447] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:40:19 (1755697219) [ 1799.470413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.612242] LustreError: 123540:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.617629] LustreError: 123540:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1799.632222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.868855] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1803.783069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1805.341336] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1807.245752] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1808.496296] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1810.736290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1812.769761] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1814.842534] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1832.636731] Lustre: *** cfs_fail_loc=159, val=0*** [ 1847.907294] Lustre: lustre-MDT0000: Client cceef0f1-b46a-497a-be4d-e2c045d584c1 (at 192.168.201.58@tcp) reconnecting [ 1848.549676] Lustre: *** cfs_fail_loc=159, val=0*** [ 1848.552032] Lustre: Skipped 5 previous similar messages [ 1863.780870] Lustre: lustre-MDT0000: Client cceef0f1-b46a-497a-be4d-e2c045d584c1 (at 192.168.201.58@tcp) reconnecting [ 1865.184914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.187312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.192390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.235314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1871.215104] LustreError: 126030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1871.218470] LustreError: 126030:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1871.285192] Lustre: server umount lustre-OST0000 complete [ 1875.936870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1875.939763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.945840] Lustre: Skipped 1 previous similar message [ 1882.593200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.596054] Lustre: Skipped 6 previous similar messages [ 1887.200119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1887.231548] LustreError: 126231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1887.233608] LustreError: 126231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1887.260310] Lustre: server umount lustre-MDT0000 complete [ 1887.712991] LustreError: 123539:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.718718] LustreError: 123539:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1888.614589] LustreError: 123519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697311 with bad export cookie 18389437486649214321 [ 1888.618405] LustreError: 123519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1888.619693] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.754955] Lustre: server umount lustre-MDT0001 complete [ 1892.882522] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1893.971756] Key type lgssc unregistered [ 1894.101335] LNet: 126962:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.104389] LNetError: 126962:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.113298] LNet: Removed LNI 192.168.201.158@tcp [ 1894.424931] Key type .llcrypt unregistered [ 1894.426289] Key type ._llcrypt unregistered [ 1903.918548] Key type ._llcrypt registered [ 1903.919838] Key type .llcrypt registered [ 1903.955903] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1904.383278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.410979] alg: No test for adler32 (adler32-zlib) [ 1905.268667] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1905.354543] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1906.936557] Key type lgssc registered [ 1907.322188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1910.009797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1910.017027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.190429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.583924] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1915.685679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1915.824492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1917.204028] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1919.115824] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1920.380159] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.649986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1922.768267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.640518] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1926.449763] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1936.103158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 1941.530019] Lustre: *** cfs_fail_loc=159, val=0*** [ 1956.966600] Lustre: lustre-MDT0000: Client debff038-6e56-4572-a5d9-d4527a01f601 (at 192.168.201.58@tcp) reconnecting [ 1957.860591] Lustre: *** cfs_fail_loc=159, val=0*** [ 1974.371611] Lustre: lustre-MDT0001: Client debff038-6e56-4572-a5d9-d4527a01f601 (at 192.168.201.58@tcp) reconnecting [ 1974.931564] Lustre: *** cfs_fail_loc=159, val=0*** [ 1974.933465] Lustre: Skipped 3 previous similar messages [ 1991.269398] Lustre: lustre-MDT0001: Client debff038-6e56-4572-a5d9-d4527a01f601 (at 192.168.201.58@tcp) reconnecting [ 1992.660296] Lustre: 131032:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1993.604651] Lustre: *** cfs_fail_loc=159, val=0*** [ 1993.606598] Lustre: Skipped 3 previous similar messages [ 2009.192496] Lustre: lustre-MDT0000: Client 42e82473-c7e8-4242-9300-71df0d7bae72 (at 192.168.201.58@tcp) reconnecting [ 2009.799353] Lustre: *** cfs_fail_loc=159, val=0*** [ 2009.801196] Lustre: Skipped 12 previous similar messages [ 2026.080606] Lustre: lustre-MDT0000: Client 42e82473-c7e8-4242-9300-71df0d7bae72 (at 192.168.201.58@tcp) reconnecting [ 2026.599145] Lustre: 131325:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2028.515529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2028.516321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.517272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.518347] LustreError: Skipped 1 previous similar message [ 2028.523494] Lustre: Skipped 1 previous similar message [ 2033.635025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2033.638054] Lustre: Skipped 2 previous similar messages [ 2034.030911] LustreError: 131422:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2034.090673] Lustre: server umount lustre-OST0000 complete [ 2038.753696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2038.753958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.754737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.765968] Lustre: Skipped 1 previous similar message [ 2041.533542] LustreError: 131624:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2041.536269] LustreError: 131624:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2041.567584] Lustre: server umount lustre-MDT0000 complete [ 2042.883198] LustreError: 131152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697465 with bad export cookie 2093219125873663555 [ 2042.887062] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.887846] LustreError: 131152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.957106] LustreError: 131826:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2042.960380] LustreError: 131826:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2043.041124] Lustre: server umount lustre-MDT0001 complete [ 2046.934292] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2048.044101] Key type lgssc unregistered [ 2048.173334] LNet: 132352:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.176942] LNetError: 132352:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.185330] LNet: Removed LNI 192.168.201.158@tcp [ 2048.504864] Key type .llcrypt unregistered [ 2048.506558] Key type ._llcrypt unregistered [ 2057.720805] Key type ._llcrypt registered [ 2057.722415] Key type .llcrypt registered [ 2057.758975] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2058.056279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.103098] alg: No test for adler32 (adler32-zlib) [ 2058.946309] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2059.018584] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2060.600180] Key type lgssc registered [ 2060.958981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2063.441309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2063.448806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.610587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.908556] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2068.766290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2068.903589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.303702] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2072.154746] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2073.474399] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2075.856323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2075.993167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.069338] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2079.080934] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2080.014339] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2090.348565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2092.718920] Lustre: 135972:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2093.353289] Lustre: 136020:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2093.834414] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2094.944773] Lustre: 136169:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2095.246587] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2095.249362] Lustre: Skipped 1 previous similar message [ 2095.549881] Lustre: 136267:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2099.683449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2099.685174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.690108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2100.707920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2100.708470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.709884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2102.572243] LustreError: 136364:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2102.648473] Lustre: server umount lustre-OST0000 complete [ 2104.800980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2104.804272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.809881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.812711] Lustre: Skipped 1 previous similar message [ 2110.079385] LustreError: 136573:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2110.082333] LustreError: 136573:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2110.112465] Lustre: server umount lustre-MDT0000 complete [ 2110.946183] LustreError: 133718:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.950772] LustreError: 133718:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2111.370035] LustreError: 133697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697534 with bad export cookie 7029149670618704225 [ 2111.373446] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.374301] LustreError: 133697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2111.445103] LustreError: 136774:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2111.448230] LustreError: 136774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2111.550584] Lustre: server umount lustre-MDT0001 complete [ 2114.968800] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2116.111105] Key type lgssc unregistered [ 2116.238474] LNet: 137299:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.241891] LNetError: 137299:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.252424] LNet: Removed LNI 192.168.201.158@tcp [ 2116.569445] Key type .llcrypt unregistered [ 2116.571061] Key type ._llcrypt unregistered [ 2125.276299] Key type ._llcrypt registered [ 2125.277265] Key type .llcrypt registered [ 2125.312750] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2125.708001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2125.751636] alg: No test for adler32 (adler32-zlib) [ 2126.602791] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2126.677095] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2128.256159] Key type lgssc registered [ 2128.651097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2131.197890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2131.204159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.369944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.636500] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2136.435217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2136.561507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2137.877646] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2139.743763] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2140.953674] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.319979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2143.453875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.509458] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2146.535536] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2147.359503] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2157.797777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2164.876952] Lustre: *** cfs_fail_loc=159, val=0*** [ 2181.216238] Lustre: lustre-MDT0000: Client 994b1919-e297-4600-bc92-9badadc69f93 (at 192.168.201.58@tcp) reconnecting [ 2182.626471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.629478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.633664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2183.650822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2183.651112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.651193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.653138] Lustre: Skipped 1 previous similar message [ 2188.653105] LustreError: 141046:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2188.722744] Lustre: server umount lustre-OST0000 complete [ 2192.865142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.868538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.874696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2196.032385] LustreError: 141249:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2196.034467] LustreError: 141249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2196.058688] Lustre: server umount lustre-MDT0000 complete [ 2197.257546] LustreError: 138645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697619 with bad export cookie 5820690444186177863 [ 2197.260853] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.262442] LustreError: 138645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2197.332433] LustreError: 141451:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2197.334847] LustreError: 141451:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2197.420456] Lustre: server umount lustre-MDT0001 complete [ 2200.892129] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2202.039845] Key type lgssc unregistered [ 2202.173422] LNet: 141977:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.176637] LNetError: 141977:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.185349] LNet: Removed LNI 192.168.201.158@tcp [ 2202.521302] Key type .llcrypt unregistered [ 2202.522497] Key type ._llcrypt unregistered [ 2211.503919] Key type ._llcrypt registered [ 2211.505583] Key type .llcrypt registered [ 2211.546144] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2211.880280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2211.938800] alg: No test for adler32 (adler32-zlib) [ 2212.797114] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2212.892050] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2214.480126] Key type lgssc registered [ 2214.834787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2217.426390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2217.433048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.607067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.937716] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2222.853366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2222.996385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.322061] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2226.177567] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2227.377102] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2229.682106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2229.825710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.800307] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2232.935831] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2233.853267] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2244.207204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2251.757444] Lustre: lustre-OST0000: evicting e45d0c4f-194e-4c90-84be-4ed2ed3b04c3 (at 192.168.201.58@tcp) by administrative request [ 2251.760428] Lustre: Skipped 1 previous similar message [ 2279.394022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.397212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.403981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.418629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.419023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.419249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2280.425884] Lustre: Skipped 1 previous similar message [ 2282.032105] LustreError: 146108:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2282.098745] Lustre: server umount lustre-OST0000 complete [ 2284.512876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.515196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.519316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.469401] LustreError: 146310:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2289.471392] LustreError: 146310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2289.499952] Lustre: server umount lustre-MDT0000 complete [ 2290.656847] LustreError: 144069:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.662346] LustreError: 144069:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2290.691343] LustreError: 143323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697713 with bad export cookie 8253048652795476409 [ 2290.694248] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.694727] LustreError: 143323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2290.767987] LustreError: 146512:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2290.769657] LustreError: 146512:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2290.842084] Lustre: server umount lustre-MDT0001 complete [ 2294.200470] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2295.322518] Key type lgssc unregistered [ 2295.479669] LNet: 147038:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.482871] LNetError: 147038:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.493470] LNet: Removed LNI 192.168.201.158@tcp [ 2295.834049] Key type .llcrypt unregistered [ 2295.835694] Key type ._llcrypt unregistered [ 2319.231325] Key type ._llcrypt registered [ 2319.232493] Key type .llcrypt registered [ 2319.267479] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2319.708941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2319.720669] alg: No test for adler32 (adler32-zlib) [ 2320.570794] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2320.650900] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2322.224124] Key type lgssc registered [ 2322.547735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2325.048878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2325.054991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.201716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.406923] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2330.140724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2330.266386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2331.473286] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2333.213986] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2334.406975] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.551402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2336.672027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.459296] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2339.750928] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2340.208475] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2351.014609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2360.288713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.290874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.294783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.314328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.316748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.316753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.397079] LustreError: 151732:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2364.456320] Lustre: server umount lustre-OST0000 complete [ 2366.435581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2366.435612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.442741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.444357] Lustre: Skipped 2 previous similar messages [ 2371.553878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.555943] Lustre: Skipped 1 previous similar message [ 2371.770180] LustreError: 151933:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2371.771918] LustreError: 151933:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2371.800830] Lustre: server umount lustre-MDT0000 complete [ 2372.977216] LustreError: 149452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697795 with bad export cookie 18124587558648092924 [ 2372.979813] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2372.981610] LustreError: 149452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2373.037838] LustreError: 152134:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2373.040016] LustreError: 152134:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2373.112769] Lustre: server umount lustre-MDT0001 complete [ 2376.571125] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2377.595709] Key type lgssc unregistered [ 2377.727378] LNet: 152661:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2377.729814] LNetError: 152661:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.737315] LNet: Removed LNI 192.168.201.158@tcp [ 2378.031044] Key type .llcrypt unregistered [ 2378.031964] Key type ._llcrypt unregistered [ 2386.644204] Key type ._llcrypt registered [ 2386.645239] Key type .llcrypt registered [ 2386.679597] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2387.078264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2387.088694] alg: No test for adler32 (adler32-zlib) [ 2387.979601] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2388.075199] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2389.664235] Key type lgssc registered [ 2389.993282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2392.279840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2394.877472] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 09:50:16 (1755697816) [ 2399.772458] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 2402.715132] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2405.791583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2407.960873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2409.983092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2412.010085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2414.211810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2414.234877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2414.242038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.322576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2415.335546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2415.369256] Lustre: lustre-MDT0000: new disk, initializing [ 2415.394470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.400797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2416.673122] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2420.804028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2420.828333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2420.864612] Lustre: 156840:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2420.888360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2420.890305] Lustre: Skipped 1 previous similar message [ 2420.925767] Lustre: lustre-MDT0001: new disk, initializing [ 2420.961155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2420.972280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2420.975781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2422.156557] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2425.125693] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2426.228635] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2428.446112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2428.470583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2428.574914] Lustre: lustre-OST0000: new disk, initializing [ 2428.577621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2428.606546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.380951] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2431.284798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2431.289207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2431.316437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2433.301972] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2436.578091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2436.578419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.582151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2436.585441] Lustre: Skipped 1 previous similar message [ 2441.701432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.703211] Lustre: Skipped 2 previous similar messages [ 2442.413889] LustreError: 159407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2442.472588] Lustre: server umount lustre-OST0000 complete [ 2446.818369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.818382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2446.819144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2446.821870] Lustre: Skipped 1 previous similar message [ 2446.827604] LustreError: Skipped 1 previous similar message [ 2449.790776] LustreError: 159608:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2449.793650] LustreError: 159608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2449.833500] Lustre: server umount lustre-MDT0000 complete [ 2451.056916] LustreError: 156831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697873 with bad export cookie 8825586699520138717 [ 2451.058887] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2451.060010] LustreError: 156831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2451.118247] LustreError: 159810:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2451.120112] LustreError: 159810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2451.193810] Lustre: server umount lustre-MDT0001 complete [ 2454.899631] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2455.874819] Key type lgssc unregistered [ 2456.006487] LNet: 160336:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.008873] LNetError: 160336:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2456.017282] LNet: Removed LNI 192.168.201.158@tcp [ 2456.307612] Key type .llcrypt unregistered [ 2456.308942] Key type ._llcrypt unregistered [ 2465.369401] Key type ._llcrypt registered [ 2465.370824] Key type .llcrypt registered [ 2465.409163] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2465.785989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2465.837967] alg: No test for adler32 (adler32-zlib) [ 2466.704374] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2466.787632] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2468.376157] Key type lgssc registered [ 2468.762308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2471.481067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2471.488962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.652219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.005792] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2477.147769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2477.293076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2478.621476] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2480.366768] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2481.517416] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2483.758862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2483.870735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.696017] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2487.380772] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2498.216349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2505.526121] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2505.532115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.536631] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2505.538844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.543442] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2505.837121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.146631] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2506.149389] Lustre: Skipped 1 previous similar message [ 2506.154730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.158208] Lustre: Skipped 1 previous similar message [ 2506.159594] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2506.161363] Lustre: Skipped 1 previous similar message [ 2506.162655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.166659] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2506.168788] Lustre: Skipped 1 previous similar message [ 2506.487245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.778192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.063490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.347740] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2507.349806] Lustre: Skipped 3 previous similar messages [ 2507.355259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.360049] Lustre: Skipped 3 previous similar messages [ 2507.361375] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2507.363344] Lustre: Skipped 3 previous similar messages [ 2507.364587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.367378] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2507.369607] Lustre: Skipped 3 previous similar messages [ 2507.636368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.914712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.213252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.544421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.847423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.152383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.454115] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2509.456283] Lustre: Skipped 6 previous similar messages [ 2509.462016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.466704] Lustre: Skipped 6 previous similar messages [ 2509.468571] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2509.471026] Lustre: Skipped 6 previous similar messages [ 2509.472593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.476562] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2509.479013] Lustre: Skipped 6 previous similar messages [ 2509.786725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.083740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.376573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.683804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.014283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.316613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.642399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.945926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.284146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.600647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.906427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.211390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.521600] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2513.524251] Lustre: Skipped 12 previous similar messages [ 2513.526714] LustreError: 161720:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2513.531397] LustreError: 161720:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2513.531666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.539057] Lustre: Skipped 12 previous similar messages [ 2513.541141] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2513.544138] Lustre: Skipped 12 previous similar messages [ 2513.545468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.549123] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2513.551212] Lustre: Skipped 12 previous similar messages [ 2513.861587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.173132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.491250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.830609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.161092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.483696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.791817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.114289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.429458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.740032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.046371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.352761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.650425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.971185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.267957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.576289] LustreError: 161720:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2518.579300] LustreError: 161720:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2518.580399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.878921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.180788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.480715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.799204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.102362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.408584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.705049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.020473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.316109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.607757] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2521.609627] Lustre: Skipped 25 previous similar messages [ 2521.614986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.618591] Lustre: Skipped 25 previous similar messages [ 2521.620110] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2521.622161] Lustre: Skipped 25 previous similar messages [ 2521.623525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.628151] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2521.632263] Lustre: Skipped 25 previous similar messages [ 2521.916928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.232718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.532749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.832134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.135567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.464585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.783254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.086769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.391141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.723663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.030571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.327329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.633745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.947045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.245075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.543313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.835146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.152556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.461103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.766466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.070537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.380667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.679437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.976598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.268579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.584842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.893740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.206071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.509862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.827309] LustreError: 161720:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2530.829838] LustreError: 161720:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2530.831376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.141237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.444086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.742428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.063053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.365199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.668952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.968254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.271834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.562911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.865971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.162104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.479906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.791221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.097439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.407993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.722889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.031188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.353558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.654472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.980126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.296431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.610907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.923093] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2537.924969] Lustre: Skipped 52 previous similar messages [ 2537.931220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.936308] Lustre: Skipped 52 previous similar messages [ 2537.937704] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2537.939757] Lustre: Skipped 52 previous similar messages [ 2537.941394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.944935] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2537.947557] Lustre: Skipped 52 previous similar messages [ 2538.273093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.607256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.914396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.229115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.564963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.909118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.239339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.568119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.896351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.205070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.527695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.842813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.168476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.478983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.793944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.102028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.446225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.773867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.095615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.407735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.748358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.059822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.367190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.677730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.010028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.324857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.639784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.952620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.268944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.571893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.892607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.207021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.525566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.849190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.172637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.482868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.815255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.137966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.478880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.815661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.159498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.483365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.806714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.125320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.464727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.785895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.092136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.411722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.750616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.057251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.373956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.690869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.012973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.320742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.639751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.958705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.298985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.609159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.913598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.221535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.546693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.875404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.183347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.488882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.821428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.127219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.429364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.730712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.053307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.367285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.680325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.993582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.324928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.625381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.951221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.263802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.599079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.911553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.213100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.539645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.881465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.191275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.523559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.862923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.246909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.594105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.941853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.275204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.623724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.927928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.221414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.520320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.827114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.119529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.440656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.756398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.070096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.375327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.670165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.953559] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2569.955588] Lustre: Skipped 99 previous similar messages [ 2569.961254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.964684] Lustre: Skipped 99 previous similar messages [ 2569.966594] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2569.968478] Lustre: Skipped 99 previous similar messages [ 2569.970699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.974405] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2569.977293] Lustre: Skipped 99 previous similar messages [ 2570.260310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.559328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.861809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.163642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.482275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.777933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.076082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.368593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.686966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.988785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.277697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.554096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.855178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.144972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.447098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.733674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.038907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.347962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.643165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.931926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.253891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.568627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.865522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.158847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.484202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.764668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.046159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.337418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.639590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.930308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.231558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.510766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.801221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.090543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.391384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.670979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.970291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.255348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.536116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.834865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.174524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.480939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.811396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.117464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.440584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.726241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.015388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.297036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.599879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.888635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.178436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.496492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.825727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.121651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.423408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.723301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.063331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.380109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.692653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.984239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.279048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.576938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.863794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.171719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.515927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.830629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.149686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.446306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.757635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.036713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.321996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.612429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.905673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.195914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.510101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.807767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.151061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.475925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.780262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.058808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.339640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.633281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.953201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2595.245344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2595.560304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2595.846903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.152853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.452172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.787794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.107095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.430990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.745132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.052698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.376402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.702868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.022615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.361650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.677540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.999910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.292525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.610960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.932629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.267036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.599255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.000850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.310789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.600772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.899571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.198808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.497880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.829236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.139877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.451796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.745542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.075693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.400593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.705923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.007424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.308573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.602675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.904852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.196844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.473655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.773202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.086837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.406161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.717218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.029416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.342797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.628282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.917530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.211896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.534306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.863933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.169403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.461560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.770129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.080883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.379378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.663630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.977417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.281846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.608188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.904351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.229263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.515403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.810868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.113578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.421124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.702877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.005538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.305842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.595032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.875236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.176654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.478650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.778846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.066474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.364821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.679622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.007699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.318530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.623852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.919584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.224204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.506910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.800323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.110180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.444453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.745269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.027254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.318296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.621721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.910888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.200791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.483303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.774273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.053184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.340603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.614945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.915518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.228548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.519194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.814927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.132134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.436437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.741891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.025076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.315683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.610350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.891506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.182922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.483460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.795040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.102939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.397992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.723538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.028923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.330890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.628655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.944867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.239246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.537601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.825831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.129157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.428239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.734911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.024375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.319626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.622018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.933494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.246032] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2634.248155] Lustre: Skipped 211 previous similar messages [ 2634.254174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.257871] Lustre: Skipped 211 previous similar messages [ 2634.259808] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2634.263267] Lustre: Skipped 211 previous similar messages [ 2634.265991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.270615] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2634.274631] Lustre: Skipped 211 previous similar messages [ 2634.603275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.880191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.160901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.443504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.741207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.035655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.338326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.631248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.934276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.245297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.534264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.821791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2638.133524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2638.422803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2638.722171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.008913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.314187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.591835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.876364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.163973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.461313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.768666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.083122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.385447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.706039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.992252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.277519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.560449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.861733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.169040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.450458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.733136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.047146] LustreError: 161720:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2644.049826] LustreError: 161720:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2644.051135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.339852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.633964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.934912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.240109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.531984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.817182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.112085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.416326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.697027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.983149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2647.265151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2647.550902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2647.834142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.117941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.401940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.707403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.002762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.307820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.589156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.897731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2650.177277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2650.452955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2650.737273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2651.029802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2651.309909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2651.603787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2651.909650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2652.215936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2652.512794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2652.831301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2653.154811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2653.447408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2653.748486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2654.037480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2654.317440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2654.624942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2654.909408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2655.204492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2655.502097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2655.824632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2656.116816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2656.432024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2656.717588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2657.021463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2657.326814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2657.627359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2657.932799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2658.230097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2658.520538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2658.830199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2659.144904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2659.488038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2659.798579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2660.083988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2660.371937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2660.673028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2660.954491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2661.232292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2661.517278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2661.837568] LustreError: 161720:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2661.841056] LustreError: 161720:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2661.843381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2662.159691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2662.481239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2662.804635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2663.140932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2663.463320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2663.791562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2664.105214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2664.467097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2664.779440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2665.100948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2665.416161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2665.745364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2666.065877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2666.364963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2666.672428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2667.006338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2667.339846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2667.670341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2668.001683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2668.354970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2668.675396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2668.993762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2669.297570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2669.625690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2669.942089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2670.281991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2670.601756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2670.946408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2671.265634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2671.580229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2671.883242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2672.221793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2672.553307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2672.856931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2673.153516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2673.493307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2673.787859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2674.096370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2674.394780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2674.712782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2675.007520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2675.307782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2675.608863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2675.931180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2676.230698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2676.517585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2676.814759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2677.148856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2677.474488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2677.801655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2678.130812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2678.481847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2678.815479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2679.144312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2679.472913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2679.814324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2680.113104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2680.425893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2680.747801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2681.095179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2681.421625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2681.743316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2682.069881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2682.384338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2682.694377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2683.006695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2683.318600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2683.642350] LustreError: 161720:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2683.646490] LustreError: 161720:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2683.647535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2683.948711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2684.287099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2684.606246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2684.934766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2685.249343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2685.572715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2685.878379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2686.191245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2686.503966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2686.806632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2687.123796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2687.462511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2687.758522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2688.085903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2688.381480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2688.680655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2689.053470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2689.379750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2689.681914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2689.991866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2690.354103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2690.669881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2690.981831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2691.298421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2691.601604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2691.916274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2692.216657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2692.518052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2692.828966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2693.157683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2693.485934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2693.813557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2694.135530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2694.491492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2694.803679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2695.117068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2695.437271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2695.767103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2696.089197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2696.404969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2696.719399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2697.081908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2697.400700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2697.715054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2698.029596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2698.346128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2698.658703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2698.980395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2699.298889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2699.644395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2699.952481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2700.271363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2700.580549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2700.912350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2701.216899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2701.535907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2701.845193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2702.176532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2702.492218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2702.797222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2703.100524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2703.437410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2703.742639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2704.055974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2704.349862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2704.666449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2704.968587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2705.276469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2705.589656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2705.905639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2706.205587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2706.526963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2706.837118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2707.172606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2707.499796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2707.813253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2708.124707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2708.451120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2708.756709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.090369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.416955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.777023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2710.108972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2710.426263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2710.751576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.086279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.410185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.740560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.055038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.384856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.710744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.060112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.395187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.755146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.071737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.393927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.722096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2715.054637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2715.367414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2715.688572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.002864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.336162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.651989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.976874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.294163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.628034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.939714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.253605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.585418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.925428] LustreError: 161720:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2718.929468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.929678] LustreError: 161720:0:(osp_precreate.c:654:osp_precreate_send()) Skipped 1 previous similar message [ 2718.937234] LustreError: 161720:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2718.942760] LustreError: 161720:0:(osp_precreate.c:1358:osp_precreate_thread()) Skipped 1 previous similar message [ 2719.278316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.652491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.987866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.309095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.627230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.983477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.317929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.643032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.977933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.308441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.631972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.385751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2745.390780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.391009] LustreError: Skipped 5 previous similar messages [ 2748.386438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2748.388238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.389362] LustreError: Skipped 1 previous similar message [ 2751.555079] LustreError: 200071:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2751.669060] Lustre: server umount lustre-OST0000 complete [ 2753.506959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2753.509120] Lustre: Skipped 2 previous similar messages [ 2758.625161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.627726] Lustre: Skipped 1 previous similar message [ 2763.745159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.748451] Lustre: Skipped 1 previous similar message [ 2767.328117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2767.364745] LustreError: 200272:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2767.366459] LustreError: 200272:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2767.420436] Lustre: server umount lustre-MDT0000 complete [ 2768.666338] LustreError: 161682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698191 with bad export cookie 1317345133064700466 [ 2768.668712] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.669778] LustreError: 161682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2768.735113] LustreError: 200474:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2768.737898] LustreError: 200474:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2768.821295] Lustre: server umount lustre-MDT0001 complete [ 2772.654826] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2773.780661] Key type lgssc unregistered [ 2773.909396] LNet: 201000:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2773.912503] LNetError: 201000:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2773.920375] LNet: Removed LNI 192.168.201.158@tcp [ 2774.215308] Key type .llcrypt unregistered [ 2774.216407] Key type ._llcrypt unregistered [ 2783.576969] Key type ._llcrypt registered [ 2783.577979] Key type .llcrypt registered [ 2783.615613] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2783.969464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2784.027423] alg: No test for adler32 (adler32-zlib) [ 2784.878456] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2784.966093] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2786.544172] Key type lgssc registered [ 2786.909504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2789.485696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2789.492515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.644906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.977211] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2794.938251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2795.072702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2796.443173] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2798.349325] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2799.642528] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2802.034102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2802.160622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.107260] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2805.916446] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2815.465869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15347 to 0x280000401:15393) [ 2819.559188] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2820.223085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.227931] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2820.230149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.233213] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2820.233224] LustreError: 202383:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2821.430408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2821.430523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.432504] LustreError: Skipped 2 previous similar messages [ 2821.436312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2825.699275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2825.701399] Lustre: Skipped 1 previous similar message [ 2825.701599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.702054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2826.835278] Lustre: lustre-OST0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 2826.837705] Lustre: Skipped 1 previous similar message [ 2827.502213] LustreError: 204767:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2827.584065] Lustre: server umount lustre-OST0000 complete [ 2840.312148] LustreError: 202383:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2855.743938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2855.860572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.765431] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2859.562333] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2866.084601] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.087911] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2876.325410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.344360] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.158@tcp (at 0@lo) [ 2879.609598] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 18 sec [ 2881.332225] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a7007241000.ost_server_uuid 50 [ 2881.829872] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a7007241000.ost_server_uuid in FULL state after 0 sec [ 2886.625431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2886.628224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.632263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2886.635432] Lustre: Skipped 1 previous similar message [ 2888.876109] LustreError: 206112:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2888.878154] LustreError: 206112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2888.945418] Lustre: server umount lustre-OST0000 complete [ 2891.746575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.747771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.750179] Lustre: Skipped 2 previous similar messages [ 2896.442915] LustreError: 206313:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2896.444711] LustreError: 206313:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2896.471823] Lustre: server umount lustre-MDT0000 complete [ 2896.864954] LustreError: 202364:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.869121] LustreError: 202364:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2897.711270] LustreError: 202346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698320 with bad export cookie 5465135519462270169 [ 2897.713666] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.715804] LustreError: 202346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2897.858599] Lustre: server umount lustre-MDT0001 complete [ 2901.618369] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2902.712863] Key type lgssc unregistered [ 2902.848361] LNet: 207041:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2902.850522] LNetError: 207041:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2902.859300] LNet: Removed LNI 192.168.201.158@tcp [ 2903.155803] Key type .llcrypt unregistered [ 2903.156867] Key type ._llcrypt unregistered [ 2914.034735] Key type ._llcrypt registered [ 2914.036699] Key type .llcrypt registered [ 2914.075956] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2914.375588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2914.385757] alg: No test for adler32 (adler32-zlib) [ 2915.245672] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2915.329136] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2916.912279] Key type lgssc registered [ 2917.280406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2920.366311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2920.372571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.531314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.811272] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2925.902248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2926.031357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2927.515169] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2928.534336] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2930.851976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2930.975283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.010059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15395 to 0x280000401:15425) [ 2933.697444] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2937.665814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2937.692733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2937.750206] Lustre: lustre-OST0001: new disk, initializing [ 2937.753516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2937.793836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2938.880551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2938.883556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2938.919204] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2939.726339] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2945.218212] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2950.056731] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2961.890564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2961.894975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.901602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.451251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.452713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.455962] Lustre: Skipped 1 previous similar message [ 2964.459590] Lustre: Skipped 1 previous similar message [ 2967.489293] LustreError: 212050:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2967.518837] Lustre: server umount lustre-MDT0000 complete [ 2968.732355] LustreError: 208701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698391 with bad export cookie 1744031661738135174 [ 2968.736411] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2968.737571] LustreError: 208701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2968.807677] LustreError: 212251:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2968.811402] LustreError: 212251:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2968.895738] Lustre: server umount lustre-MDT0001 complete [ 2979.380108] LustreError: 212451:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2979.382554] LustreError: 212451:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2979.399221] Lustre: server umount lustre-OST0000 complete [ 2990.899486] LustreError: 212653:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2990.901396] LustreError: 212653:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2990.985225] Lustre: server umount lustre-OST0001 complete [ 2992.002952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2992.029810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.380784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2992.407136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.754189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2992.783209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.147315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2993.172544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2997.908483] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3001.387068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.536215] LustreError: 214055:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3001.546229] LustreError: 214055:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3001.562034] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.795855] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3005.621209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3007.034566] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3008.024828] Lustre: 215153:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3010.213098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3010.323470] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.326534] Lustre: Skipped 1 previous similar message [ 3011.364545] LustreError: 215508:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.368207] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15427 to 0x280000401:15457) [ 3012.255275] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3015.141470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3016.234782] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3017.089722] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3020.250506] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3021.447476] Lustre: 216987:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3031.526540] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.530043] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.530047] Lustre: Skipped 3 previous similar messages [ 3031.531620] Lustre: Skipped 1 previous similar message [ 3031.534100] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3036.643384] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.645290] Lustre: Skipped 4 previous similar messages [ 3036.865145] LustreError: 217293:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3036.866878] LustreError: 217293:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3036.898341] Lustre: server umount mylustre-MDT0000 complete [ 3038.139919] LustreError: 214029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698460 with bad export cookie 1744031661738137967 [ 3038.146275] LustreError: 214029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3038.146868] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3038.315747] Lustre: server umount mylustre-MDT0001 complete [ 3049.976162] LustreError: 217707:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3049.979047] LustreError: 217707:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3049.998113] Lustre: server umount mylustre-OST0000 complete [ 3060.464797] Lustre: server umount mylustre-OST0001 complete [ 3061.501608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3061.529619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3061.894359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3061.926296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3062.259623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3062.292975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3062.635117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3062.661889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3067.509081] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3071.180711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.342205] LustreError: 219309:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.365069] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.368100] Lustre: Skipped 1 previous similar message [ 3072.714841] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3075.597330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3076.970145] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3077.926189] Lustre: 220405:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3080.205035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3081.381536] LustreError: 220762:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.387690] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15459 to 0x280000401:15489) [ 3082.335447] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3085.242332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3086.389169] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 3087.236378] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3090.418035] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3091.602219] Lustre: 222240:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3101.667357] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.670320] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.670572] Lustre: Skipped 2 previous similar messages [ 3101.672144] Lustre: Skipped 1 previous similar message [ 3101.673337] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3104.637923] LustreError: 222491:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3104.640522] LustreError: 222491:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3104.672097] Lustre: server umount tfs-MDT0000 complete [ 3105.912070] LustreError: 222245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698528 with bad export cookie 1744031661738139857 [ 3105.916893] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.070090] Lustre: server umount tfs-MDT0001 complete [ 3117.634750] Lustre: server umount tfs-OST0000 complete [ 3129.200771] Lustre: server umount tfs-OST0001 complete [ 3130.221952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3130.251507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3130.614898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3130.647044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3131.001970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3131.024113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3131.377343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3131.410122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3136.263577] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3139.887400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.035873] LustreError: 224497:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.057199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.058810] Lustre: Skipped 3 previous similar messages [ 3141.362937] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3144.223350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3145.597533] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3146.569501] Lustre: 225596:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3148.764592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3149.923521] LustreError: 225951:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.926676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15459 to 0x280000401:15521) [ 3150.879161] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3153.814849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3154.929136] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 3155.827470] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3158.996101] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3160.099075] Lustre: 227430:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3167.188201] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:03:09 (1755698589) [ 3170.278284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.281222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3170.281611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.281616] Lustre: Skipped 1 previous similar message [ 3170.282110] Lustre: Skipped 2 previous similar messages [ 3174.592059] LustreError: 227777:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3174.594852] LustreError: 227777:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3174.627124] Lustre: server umount lustre-MDT0000 complete [ 3175.395301] LustreError: 224497:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.400623] LustreError: 224497:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3175.788523] LustreError: 227432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698598 with bad export cookie 1744031661738141523 [ 3175.791337] LustreError: 227432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3175.794329] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3180.514128] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.517403] Lustre: Skipped 3 previous similar messages [ 3190.240113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3190.349576] Lustre: server umount lustre-MDT0001 complete [ 3197.753183] Lustre: server umount lustre-OST0000 complete [ 3213.280127] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3213.371073] Lustre: server umount lustre-OST0001 complete [ 3215.598494] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 3218.959181] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3222.441621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3224.719187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3226.851273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3229.035739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3231.339492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3231.363580] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3231.363580] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3231.363580] [ 3231.371607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3231.456494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3231.470970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3231.515036] Lustre: lustre-MDT0000: new disk, initializing [ 3231.542453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.545849] Lustre: Skipped 3 previous similar messages [ 3231.551864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3232.902267] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3237.189802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3237.214190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3237.246403] Lustre: 230556:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3237.271729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3237.273059] Lustre: Skipped 1 previous similar message [ 3237.307318] Lustre: lustre-MDT0001: new disk, initializing [ 3237.352256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3237.355086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3238.629928] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3242.776175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3242.808555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3242.910603] Lustre: lustre-OST0000: new disk, initializing [ 3242.912823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3244.763930] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3244.795234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3244.800207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3244.834928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3248.937169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3248.956785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3249.008643] Lustre: lustre-OST0001: new disk, initializing [ 3249.010380] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3250.563598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3250.567428] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3250.600320] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3250.946751] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3255.616116] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:04:37 (1755698677) [ 3259.497756] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:04:41 (1755698681) [ 3265.505187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3265.505356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.507868] LustreError: Skipped 1 previous similar message [ 3265.508359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3265.512687] Lustre: Skipped 1 previous similar message [ 3265.517392] Lustre: Skipped 8 previous similar messages [ 3266.478083] LustreError: 233934:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3266.480431] LustreError: 233934:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3266.498973] Lustre: server umount lustre-OST0000 complete [ 3271.138475] LustreError: 232083:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.143576] LustreError: 232083:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3282.400121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3282.475048] Lustre: server umount lustre-MDT0000 complete [ 3283.779440] LustreError: 231331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698706 with bad export cookie 1744031661738143070 [ 3283.783665] LustreError: 231331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3283.784639] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3298.272166] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3298.385180] Lustre: server umount lustre-MDT0001 complete [ 3302.374890] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3312.304856] Lustre: server umount lustre-OST0001 complete [ 3314.308606] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 3320.466274] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3323.962544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3326.191709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3328.502810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3330.851081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3333.193250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3333.217165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.324252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3333.344835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3333.397425] Lustre: lustre-MDT0000: new disk, initializing [ 3333.431421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3334.695651] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3338.990793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3339.014847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3339.050938] Lustre: 237375:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3339.179763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3340.481966] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3343.647621] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3344.905612] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3347.351526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3347.380880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3349.308659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3349.348536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3349.472463] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3352.773415] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3359.716698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.719448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3359.719805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.719815] Lustre: Skipped 15 previous similar messages [ 3359.721412] Lustre: Skipped 5 previous similar messages [ 3359.724583] LustreError: Skipped 1 previous similar message [ 3361.468770] Lustre: server umount lustre-OST0000 complete [ 3377.120151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3377.183976] Lustre: server umount lustre-MDT0000 complete [ 3378.466273] LustreError: 239713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698801 with bad export cookie 1744031661738144547 [ 3378.468381] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3378.470090] LustreError: 239713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3378.610194] Lustre: server umount lustre-MDT0001 complete [ 3382.372077] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3383.553505] Key type lgssc unregistered [ 3383.694432] LNet: 240729:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3383.697359] LNetError: 240729:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3383.706440] LNet: Removed LNI 192.168.201.158@tcp [ 3384.032115] Key type .llcrypt unregistered [ 3384.033152] Key type ._llcrypt unregistered [ 3393.946647] Key type ._llcrypt registered [ 3393.947667] Key type .llcrypt registered [ 3393.987140] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3394.309602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3394.328057] alg: No test for adler32 (adler32-zlib) [ 3395.189080] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3395.268875] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3396.848122] Key type lgssc registered [ 3397.206146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3399.736527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3399.742877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.889452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.168384] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3405.003631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3405.127201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3406.403318] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3408.164348] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3409.446291] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3411.713277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3411.840624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.660450] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3415.438548] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3415.801882] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3420.131310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.131678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3420.135348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.509721] LustreError: 244419:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3422.572808] Lustre: server umount lustre-OST0000 complete [ 3425.249693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.250414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.252960] Lustre: Skipped 1 previous similar message [ 3430.370405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.372417] Lustre: Skipped 2 previous similar messages [ 3435.490732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3435.494832] Lustre: Skipped 1 previous similar message [ 3438.048179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3438.100870] LustreError: 244624:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3438.104962] LustreError: 244624:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3438.139531] Lustre: server umount lustre-MDT0000 complete [ 3439.585359] LustreError: 242142:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.589218] LustreError: 242142:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3439.696741] LustreError: 244322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698862 with bad export cookie 6330446864290280691 [ 3439.700969] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3439.702388] LustreError: 244322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3439.773272] LustreError: 244827:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3439.775804] LustreError: 244827:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3439.894425] Lustre: server umount lustre-MDT0001 complete [ 3443.914500] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3445.220389] Key type lgssc unregistered [ 3445.371436] LNet: 245353:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3445.374278] LNetError: 245353:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3445.384469] LNet: Removed LNI 192.168.201.158@tcp [ 3445.717222] Key type .llcrypt unregistered [ 3445.718350] Key type ._llcrypt unregistered [ 3453.008015] Key type ._llcrypt registered [ 3453.009202] Key type .llcrypt registered [ 3453.046896] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3453.457302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3453.464220] alg: No test for adler32 (adler32-zlib) [ 3454.311591] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3454.389496] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3455.968193] Key type lgssc registered [ 3456.471855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3460.100236] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3460.105484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3477.600345] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3483.744396] LustreError: 246497:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 3513.440172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.280084] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3517.175955] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3517.924487] LustreError: 247156:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3517.993830] Lustre: server umount lustre-OST0000 complete [ 3523.399185] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3524.504855] Key type lgssc unregistered [ 3524.658401] LNet: 247988:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3524.660454] LNetError: 247988:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3524.669435] LNet: Removed LNI 192.168.201.158@tcp [ 3525.035679] Key type .llcrypt unregistered [ 3525.037257] Key type ._llcrypt unregistered [ 3539.776605] Key type ._llcrypt registered [ 3539.777628] Key type .llcrypt registered [ 3539.816369] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3540.240689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3540.270232] alg: No test for adler32 (adler32-zlib) [ 3541.123088] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3541.205885] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3542.784232] Key type lgssc registered [ 3543.150766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3547.637719] loop0: detected capacity change from 0 to 204800000 [ 3547.693655] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.696940] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.833523] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.839198] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.865585] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.891376] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.907787] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.932529] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.959949] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3547.990251] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3548.352700] loop0: detected capacity change from 0 to 204800000 [ 3548.405793] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3548.689515] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.012736] loop0: detected capacity change from 0 to 204800000 [ 3549.143370] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.456672] loop0: detected capacity change from 0 to 204800000 [ 3549.657207] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.049346] loop0: detected capacity change from 0 to 204800000 [ 3550.061249] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3551.494182] loop0: detected capacity change from 0 to 204800000 [ 3551.504976] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3552.794206] loop0: detected capacity change from 0 to 204800000 [ 3552.804830] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3554.061269] loop0: detected capacity change from 0 to 204800000 [ 3554.071074] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3555.337499] loop0: detected capacity change from 0 to 204800000 [ 3555.370151] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3555.400532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3555.406346] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.426777] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3555.430174] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3555.433771] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3555.437309] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3555.440835] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3555.444242] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3555.447652] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3555.451129] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3555.454347] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3555.457647] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3555.460958] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3555.464413] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3555.468169] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3555.471590] Lustre: 251355:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3557.926148] LustreError: 251621:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3557.960869] Lustre: server umount MGS complete [ 3558.246911] loop0: detected capacity change from 0 to 204800000 [ 3558.278354] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.441833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.457842] Lustre: lustre-MDT0000: Aborting client recovery [ 3558.775421] loop1: detected capacity change from 0 to 204800000 [ 3558.826783] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3558.856046] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3558.876195] Lustre: 251789:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3558.879354] Lustre: 251789:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3558.882511] Lustre: 251789:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3558.885565] Lustre: 251789:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3558.888408] Lustre: 251789:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3558.891853] Lustre: 251789:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3558.894884] Lustre: 251789:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3558.901704] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3558.971215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3559.003080] Lustre: lustre-MDT0001: Aborting client recovery [ 3559.327311] loop2: detected capacity change from 0 to 204800000 [ 3559.377451] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3559.405474] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3559.426954] Lustre: 251875:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3559.431872] Lustre: 251875:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3559.435240] Lustre: 251875:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3559.438202] Lustre: 251875:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3559.440989] Lustre: 251875:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3559.443942] Lustre: 251875:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3559.447633] Lustre: 251875:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3559.496484] Lustre: Found index 0 for lustre-OST0000, updating log [ 3559.537932] Lustre: lustre-OST0000: Aborting recovery [ 3559.850669] loop3: detected capacity change from 0 to 204800000 [ 3559.908988] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3559.936297] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3559.950172] Lustre: 251964:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3559.953281] Lustre: 251964:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3559.957158] Lustre: 251964:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3559.960509] Lustre: 251964:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3559.963359] Lustre: 251964:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3559.967258] Lustre: 251964:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3559.970329] Lustre: 251964:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3560.010155] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3560.013349] Lustre: Skipped 1 previous similar message [ 3565.032047] LustreError: 251732:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -108 [ 3569.124327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3569.124711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3569.125282] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3569.129405] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3569.135908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3569.141624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3570.754039] LustreError: 251700:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 293us: rc = -2 [ 3574.242459] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.243607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3574.249402] Lustre: Skipped 3 previous similar messages [ 3574.252701] Lustre: Skipped 2 previous similar messages [ 3574.257420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3577.533506] LustreError: 252297:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3577.536983] LustreError: 252297:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3577.570756] Lustre: server umount lustre-MDT0000 complete [ 3577.870510] LustreError: 251896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699000 with bad export cookie 18425543593453930502 [ 3577.871843] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3577.874632] LustreError: 251896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3579.360471] LustreError: 251698:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.360782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.360975] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3579.360981] Lustre: Skipped 1 previous similar message [ 3579.367353] LustreError: 251698:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3579.371126] Lustre: Skipped 1 previous similar message [ 3582.368631] LustreError: 251698:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.368728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3582.374629] LustreError: 251698:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3582.376132] Lustre: Skipped 2 previous similar messages [ 3584.062282] LustreError: 252348:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3584.065692] LustreError: 252348:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3584.162898] Lustre: server umount lustre-MDT0001 complete [ 3590.570327] LustreError: 252408:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3590.573864] LustreError: 252408:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3590.602713] Lustre: server umount lustre-OST0000 complete [ 3596.972804] LustreError: 252459:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3596.975642] LustreError: 252459:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3597.065750] Lustre: server umount lustre-OST0001 complete [ 3599.757863] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 10:10:21 (1755699021) [ 3608.788391] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 3612.085703] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3615.707892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3617.979444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3619.968945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3622.220756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3624.791828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3624.816470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.917202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3624.933152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3624.980086] Lustre: lustre-MDT0000: new disk, initializing [ 3625.005724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.012897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3626.357938] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3630.900681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3630.935880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3630.979995] Lustre: 255991:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3631.012470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3631.014289] Lustre: Skipped 1 previous similar message [ 3631.053821] Lustre: lustre-MDT0001: new disk, initializing [ 3631.089177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3631.099794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3631.102632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3632.439433] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3635.636122] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3636.909817] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3639.690131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.728153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3639.852386] Lustre: lustre-OST0000: new disk, initializing [ 3639.855669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3639.890915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.196393] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3645.740473] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3647.498754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3647.505554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3647.548423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3654.023366] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3662.408245] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3670.181341] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3678.014353] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 3719.141894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.142339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.155556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.159582] Lustre: Skipped 1 previous similar message [ 3723.376133] LustreError: 259688:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3723.379085] LustreError: 259688:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3723.455474] Lustre: server umount lustre-OST0000 complete [ 3728.353381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3728.356933] LustreError: Skipped 1 previous similar message [ 3728.359741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.366247] Lustre: Skipped 1 previous similar message [ 3728.369609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3731.188216] Lustre: server umount lustre-MDT0000 complete [ 3732.459582] LustreError: 256768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699155 with bad export cookie 18425543593453933211 [ 3732.462299] LustreError: 256768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3732.463091] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3732.625714] Lustre: server umount lustre-MDT0001 complete [ 3736.467695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.246720] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3740.554902] LustreError: 261025:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3740.557986] LustreError: 261025:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3740.586556] Lustre: server umount MGS complete [ 3745.473440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.664348] LustreError: 261649:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3745.671898] LustreError: 261661:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3745.694495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3747.070219] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3750.881964] LustreError: 261649:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.075672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3752.882976] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3754.566913] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3755.684801] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3758.328331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3760.434268] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3762.675938] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3796.454358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.454402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3796.461684] Lustre: Skipped 1 previous similar message [ 3796.466376] LustreError: Skipped 1 previous similar message [ 3796.469965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3796.475830] Lustre: Skipped 3 previous similar messages [ 3801.456146] LustreError: 264311:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3801.457988] LustreError: 264311:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3801.529628] Lustre: server umount lustre-OST0000 complete [ 3806.692903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.699752] Lustre: Skipped 2 previous similar messages [ 3809.059606] Lustre: server umount lustre-MDT0000 complete [ 3810.622244] LustreError: 263257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699233 with bad export cookie 18425543593453935136 [ 3810.626799] LustreError: 263257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3810.627876] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3810.827411] Lustre: server umount lustre-MDT0001 complete [ 3814.560541] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3815.692762] Key type lgssc unregistered [ 3815.833428] LNet: 265240:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3815.836037] LNetError: 265240:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3815.845369] LNet: Removed LNI 192.168.201.158@tcp [ 3816.159695] Key type .llcrypt unregistered [ 3816.160721] Key type ._llcrypt unregistered [ 3828.814576] Key type ._llcrypt registered [ 3828.815792] Key type .llcrypt registered [ 3828.855265] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 3836.215547] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3836.630827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3836.680077] alg: No test for adler32 (adler32-zlib) [ 3837.537916] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3837.635189] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3839.224208] Key type lgssc registered [ 3839.676114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3842.884879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3845.122393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3847.161073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3849.357832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3851.669977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3851.689729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3851.694770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.788489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3852.800335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3852.837759] Lustre: lustre-MDT0000: new disk, initializing [ 3852.862489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.870458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3854.486469] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3858.867185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3858.899119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3858.936457] Lustre: 269134:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3858.965319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3858.967058] Lustre: Skipped 1 previous similar message [ 3859.008660] Lustre: lustre-MDT0001: new disk, initializing [ 3859.057643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.074826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3859.082305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3860.584907] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3864.466977] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3865.748314] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3868.652201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3868.692749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3868.860600] Lustre: lustre-OST0000: new disk, initializing [ 3868.865436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3868.926111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.406960] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3874.523178] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3877.367969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3877.372045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3877.403827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3883.277582] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3894.327392] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3902.867773] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3938.785917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3938.787756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.790358] LustreError: Skipped 1 previous similar message [ 3938.794054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3938.798757] Lustre: Skipped 1 previous similar message [ 3943.906921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3943.908412] Lustre: Skipped 1 previous similar message [ 3944.238707] LustreError: 272969:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3944.303900] Lustre: server umount lustre-OST0000 complete [ 3945.953100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3945.955280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.958769] Lustre: Skipped 1 previous similar message [ 3945.960416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.025464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.026063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.552122] LustreError: 273170:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3951.553666] LustreError: 273170:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3951.581211] Lustre: server umount lustre-MDT0000 complete [ 3952.716341] LustreError: 269124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699375 with bad export cookie 15505964613120756684 [ 3952.719533] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3952.721965] LustreError: 269124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3952.784396] LustreError: 273372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3952.786344] LustreError: 273372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3952.877850] Lustre: server umount lustre-MDT0001 complete [ 3955.515331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.749223] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3958.334651] LustreError: 274307:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3958.337335] LustreError: 274307:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3958.363150] Lustre: server umount MGS complete [ 3962.095405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.258723] LustreError: 274942:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3962.283456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.534928] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3966.341828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3966.466906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3967.727701] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3969.476272] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3970.633382] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3972.895771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3974.788951] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3976.482834] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4013.029855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.030004] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.035217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4013.038057] LustreError: Skipped 1 previous similar message [ 4013.041793] Lustre: Skipped 2 previous similar messages [ 4015.919170] LustreError: 277590:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4015.922029] LustreError: 277590:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4015.993751] Lustre: server umount lustre-OST0000 complete [ 4017.633061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4017.634881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.637605] Lustre: Skipped 1 previous similar message [ 4023.265767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.267404] Lustre: Skipped 4 previous similar messages [ 4023.324812] Lustre: server umount lustre-MDT0000 complete [ 4024.499282] LustreError: 274910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699447 with bad export cookie 15505964613120758497 [ 4024.504338] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4024.570734] LustreError: 277993:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4024.572559] LustreError: 277993:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4024.669297] Lustre: server umount lustre-MDT0001 complete [ 4028.079065] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4029.104677] Key type lgssc unregistered [ 4029.230400] LNet: 278520:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4029.232410] LNetError: 278520:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4029.241433] LNet: Removed LNI 192.168.201.158@tcp [ 4029.547762] Key type .llcrypt unregistered [ 4029.548983] Key type ._llcrypt unregistered [ 4036.000149] Key type ._llcrypt registered [ 4036.001189] Key type .llcrypt registered [ 4036.046181] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 4042.092495] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4042.701243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4042.721219] alg: No test for adler32 (adler32-zlib) [ 4043.625310] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4043.740358] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4045.352253] Key type lgssc registered [ 4045.972766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4049.503765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4051.925792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4055.344252] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4058.578521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4061.264739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4061.284161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4061.288629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.374662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4062.386441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4062.428433] Lustre: lustre-MDT0000: new disk, initializing [ 4062.453223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.459576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4064.110112] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4069.432166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4069.474319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4069.630146] Lustre: lustre-OST0000: new disk, initializing [ 4069.633715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4069.637075] Lustre: Skipped 1 previous similar message [ 4069.675926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.324361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4071.330230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4071.348170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4072.183247] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4271.073836] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.079195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4276.193304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4276.679072] LustreError: 283404:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4276.823785] Lustre: server umount lustre-MDT0000 complete [ 4278.093390] LustreError: 283317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699700 with bad export cookie 18247105088925895858 [ 4278.096529] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4278.097582] LustreError: 283317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4278.125509] LustreError: 283607:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4278.127171] LustreError: 283607:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4278.205773] Lustre: server umount lustre-OST0000 complete [ 4286.430194] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 4289.367766] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4293.189032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4295.395913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4297.317154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4299.511803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4301.939653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4301.964636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.050737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4302.064332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4302.102720] Lustre: lustre-MDT0000: new disk, initializing [ 4302.129572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4302.136200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4303.548277] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4307.618176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4307.646137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4307.674898] Lustre: 286632:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4307.699919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4307.701651] Lustre: Skipped 1 previous similar message [ 4307.737945] Lustre: lustre-MDT0001: new disk, initializing [ 4307.772426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4307.784201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4307.788692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4309.012135] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4311.991567] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4313.110219] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4315.125779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4315.146685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4315.251219] Lustre: lustre-OST0000: new disk, initializing [ 4315.252848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4315.287160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.178913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4317.182901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4317.225578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4317.275300] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4320.259776] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4321.504726] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4321.577921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4322.791163] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4322.870834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4325.550121] LustreError: 289538:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4325.551915] LustreError: 289538:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4325.622103] Lustre: server umount lustre-OST0000 complete [ 4327.393943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.394478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4327.397147] Lustre: Skipped 1 previous similar message [ 4332.513102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4332.514796] Lustre: Skipped 2 previous similar messages [ 4337.633718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4337.636543] Lustre: Skipped 1 previous similar message [ 4341.216120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4341.262948] LustreError: 289739:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4341.266559] LustreError: 289739:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4341.304363] Lustre: server umount lustre-MDT0000 complete [ 4342.919890] LustreError: 286624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699765 with bad export cookie 18247105088928836635 [ 4342.926477] LustreError: 286624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4342.926513] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4352.484518] LustreError: 289942:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4352.486626] LustreError: 289942:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4352.577195] Lustre: server umount lustre-MDT0001 complete [ 4353.701131] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 4354.273696] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 10:22:56 (1755699776) [ 4356.610063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.766263] LustreError: 290516:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.772692] LustreError: 290516:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4356.789630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4358.236660] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4361.602691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4363.165375] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4365.430295] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4366.859840] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4369.555743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4372.046131] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4374.253759] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4376.782923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4376.806844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4376.864492] Lustre: lustre-OST0001: new disk, initializing [ 4376.866470] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4376.900648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4376.902475] Lustre: Skipped 2 previous similar messages [ 4379.216505] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4379.867116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4379.871938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4379.905265] LustreError: 293423:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4379.909566] LustreError: 293423:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4380.963694] LustreError: 293423:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4380.966689] LustreError: 293423:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4381.991727] LustreError: 293423:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4381.997895] LustreError: 293423:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4383.373546] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4383.480362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4384.039602] LustreError: 293436:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4384.045555] LustreError: 293436:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 4384.902978] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a700860c800.ost_server_uuid 50 [ 4385.581779] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a700860c800.ost_server_uuid in FULL state after 0 sec [ 4387.119183] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a700860c800.ost_server_uuid 50 [ 4387.818981] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a700860c800.ost_server_uuid in FULL state after 0 sec [ 4388.135382] LustreError: 293436:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4388.142232] LustreError: 293436:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 4389.170353] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4402.269206] Lustre: lustre-OST0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 4403.694632] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.697795] Lustre: Skipped 1 previous similar message [ 4405.876169] LustreError: 294287:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4405.880094] LustreError: 294287:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4405.911131] Lustre: server umount lustre-OST0001 complete [ 4408.806308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4408.807530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.811485] LustreError: 292101:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.818802] Lustre: Skipped 1 previous similar message [ 4408.828092] LustreError: 292101:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4413.928961] LustreError: 293438:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.938049] LustreError: 293438:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4414.050743] Lustre: server umount lustre-OST0000 complete [ 4418.017668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4418.021885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.041400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4419.043770] Lustre: Skipped 8 previous similar messages [ 4421.424342] Lustre: server umount lustre-MDT0000 complete [ 4423.093260] LustreError: 290542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699845 with bad export cookie 18247105088928837503 [ 4423.099639] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4423.101544] LustreError: 290542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4423.193213] LustreError: 294892:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4423.197068] LustreError: 294892:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4423.282976] Lustre: server umount lustre-MDT0001 complete [ 4428.052194] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4429.608534] Key type lgssc unregistered [ 4429.742542] LNet: 295418:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4429.744763] LNetError: 295418:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4429.751475] LNet: Removed LNI 192.168.201.158@tcp [ 4430.088232] Key type .llcrypt unregistered [ 4430.090077] Key type ._llcrypt unregistered [ 4441.321273] Key type ._llcrypt registered [ 4441.322448] Key type .llcrypt registered [ 4441.368577] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4441.788282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4441.841179] alg: No test for adler32 (adler32-zlib) [ 4442.748086] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4442.869513] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4444.488130] Key type lgssc registered [ 4444.922103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4448.016814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4448.023621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4449.174686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.414588] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4452.201818] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4454.496163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4454.621578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4455.878539] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4459.240704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4459.367533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.114857] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4462.798684] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4464.484672] LustreError: 298331:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.489318] LustreError: 298331:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4465.294912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4465.367810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4467.302137] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4469.183446] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4473.836043] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4473.836644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4476.030792] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4476.110311] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4557.793303] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4557.796851] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.803700] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4560.869311] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4560.871514] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4560.873144] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4562.481762] LustreError: 300182:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4562.506628] Lustre: server umount lustre-OST0001 complete [ 4565.987516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4565.990648] LustreError: 298329:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.990963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4565.994684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4566.002668] LustreError: 298329:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4566.010069] Lustre: Skipped 1 previous similar message [ 4571.110799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.110881] LustreError: 298329:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4571.114501] Lustre: Skipped 2 previous similar messages [ 4571.125146] LustreError: 298329:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4576.227728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4576.228614] LustreError: 298331:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4576.229829] Lustre: Skipped 1 previous similar message [ 4576.233548] LustreError: 298331:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4579.296211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4579.325162] LustreError: 300383:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4579.331397] LustreError: 300383:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4579.426438] Lustre: server umount lustre-OST0000 complete [ 4581.345612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.348869] Lustre: Skipped 2 previous similar messages [ 4586.466073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.470078] Lustre: Skipped 3 previous similar messages [ 4587.159981] LustreError: 300586:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4587.162470] LustreError: 300586:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4587.200247] Lustre: server umount lustre-MDT0000 complete [ 4588.783438] LustreError: 296772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700011 with bad export cookie 2951589240909784850 [ 4588.789313] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.791172] LustreError: 296772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4589.022604] Lustre: server umount lustre-MDT0001 complete [ 4593.404506] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4594.766154] Key type lgssc unregistered [ 4594.911516] LNet: 301315:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4594.913905] LNetError: 301315:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4594.922516] LNet: Removed LNI 192.168.201.158@tcp [ 4595.257891] Key type .llcrypt unregistered [ 4595.259031] Key type ._llcrypt unregistered [ 4616.081901] Key type ._llcrypt registered [ 4616.084470] Key type .llcrypt registered [ 4616.131751] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4616.643620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4616.662544] alg: No test for adler32 (adler32-zlib) [ 4617.503767] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4617.583744] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4619.176228] Key type lgssc registered [ 4619.853698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4624.498118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4624.511083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4625.715253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.423921] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4631.394762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4631.574627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4633.348724] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4634.674931] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4637.875935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4638.090890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.884767] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4642.219105] LustreError: 305463:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.228989] LustreError: 305463:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4645.266790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4645.434330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4648.292379] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4651.260410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 4651.261374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4657.745447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4664.460560] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4677.090987] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4677.093697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4677.096295] Lustre: Skipped 3 previous similar messages [ 4677.098540] Lustre: Skipped 2 previous similar messages [ 4679.933269] LustreError: 306938:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4679.962717] Lustre: server umount lustre-MDT0000 complete [ 4681.119197] LustreError: 303535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700103 with bad export cookie 17246229040173587309 [ 4681.122209] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4681.124771] LustreError: 303535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4681.188891] LustreError: 307139:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4681.190802] LustreError: 307139:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4681.265844] Lustre: server umount lustre-MDT0001 complete [ 4692.788847] LustreError: 307339:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4692.790956] LustreError: 307339:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4692.808703] Lustre: server umount lustre-OST0000 complete [ 4703.545055] LustreError: 307541:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4703.549139] LustreError: 307541:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4703.650675] Lustre: server umount lustre-OST0001 complete [ 4711.835898] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4716.866293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.096332] LustreError: 308710:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.126081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4718.820215] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4722.146387] LustreError: 308710:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4723.048472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4724.929184] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4726.386656] Lustre: 309807:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4729.200898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4729.342288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4729.345145] Lustre: Skipped 1 previous similar message [ 4731.711472] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4732.457734] LustreError: 310166:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.968826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4738.644009] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4741.419437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4741.419984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 4747.705712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4749.198566] Lustre: 311643:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4756.962675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.965562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.968679] Lustre: Skipped 2 previous similar messages [ 4756.970455] Lustre: Skipped 1 previous similar message [ 4761.344194] LustreError: 311746:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4761.347731] LustreError: 311746:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4761.377887] Lustre: server umount lustre-MDT0000 complete [ 4762.081403] LustreError: 309419:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.085846] LustreError: 309419:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4762.624619] LustreError: 309432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700185 with bad export cookie 17246229040173588982 [ 4762.630434] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4762.781826] Lustre: server umount lustre-MDT0001 complete [ 4774.386274] LustreError: 312148:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4774.387942] LustreError: 312148:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4774.403868] Lustre: server umount lustre-OST0000 complete [ 4786.165129] Lustre: server umount lustre-OST0001 complete [ 4787.326375] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 4787.874609] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 4788.446091] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 10:30:10 (1755700210) [ 4794.830823] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4797.641910] loop0: detected capacity change from 0 to 1099511627776 [ 4797.790102] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 4797.792541] *** VALIDATE xfs *** [ 4797.797280] XFS (loop0): Mounting V5 Filesystem [ 4797.806159] XFS (loop0): Ending clean mount [ 4798.214619] loop1: detected capacity change from 0 to 18691697672192 [ 4812.397895] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4817.693592] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 4820.744544] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4824.557395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4827.665310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.312171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4832.839194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4835.966396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4836.007986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.154217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4836.173856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4836.223031] Lustre: lustre-MDT0000: new disk, initializing [ 4836.253338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4836.256408] Lustre: Skipped 1 previous similar message [ 4836.263623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4837.619785] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4842.190087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4842.214591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4842.245880] Lustre: 316700:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4842.266475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4842.267955] Lustre: Skipped 1 previous similar message [ 4842.301677] Lustre: lustre-MDT0001: new disk, initializing [ 4842.346678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4842.350964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4843.878737] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4847.615104] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4849.141528] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4852.576915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4852.613939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4852.737277] Lustre: lustre-OST0000: new disk, initializing [ 4852.740607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4854.781089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4854.786236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4854.826058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4855.394851] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4858.934748] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4860.421087] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4860.510475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4862.031180] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4862.132141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4864.999287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4864.999507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.003894] LustreError: Skipped 1 previous similar message [ 4865.004857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4865.012148] Lustre: Skipped 2 previous similar messages [ 4865.014883] Lustre: Skipped 2 previous similar messages [ 4870.118324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4870.122044] Lustre: Skipped 2 previous similar messages [ 4875.239399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4875.243032] Lustre: Skipped 1 previous similar message [ 4877.280200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4877.308218] LustreError: 319609:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4877.313076] LustreError: 319609:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4877.432173] Lustre: server umount lustre-OST0000 complete [ 4880.356635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.360448] Lustre: Skipped 1 previous similar message [ 4885.474570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4885.477980] Lustre: Skipped 3 previous similar messages [ 4885.624859] Lustre: server umount lustre-MDT0000 complete [ 4887.469420] LustreError: 316691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700310 with bad export cookie 17246229040173590536 [ 4887.474838] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4887.476221] LustreError: 316691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4887.665761] Lustre: server umount lustre-MDT0001 complete [ 4891.759849] XFS (loop0): Unmounting Filesystem [ 4893.985783] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 10:31:55 (1755700315) [ 4896.998954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4897.163127] LustreError: 320831:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4897.190596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4897.192463] Lustre: Skipped 2 previous similar messages [ 4898.583387] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4901.653746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4903.169423] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4905.039786] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4906.193253] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4908.466290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4911.052209] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4913.620466] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4932.585114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.585856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4932.590376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4932.592208] Lustre: Skipped 1 previous similar message [ 4932.596601] LustreError: Skipped 1 previous similar message [ 4934.513189] LustreError: 323180:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4934.517549] LustreError: 323180:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4934.627375] Lustre: server umount lustre-OST0000 complete [ 4937.699732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4942.656339] Lustre: server umount lustre-MDT0000 complete [ 4942.818360] LustreError: 320830:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.832506] LustreError: 320830:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4944.487960] LustreError: 321543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700367 with bad export cookie 17246229040173591418 [ 4944.494894] LustreError: 321543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4944.496409] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.674698] Lustre: server umount lustre-MDT0001 complete [ 4949.771806] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4951.245774] Key type lgssc unregistered [ 4951.411618] LNet: 324108:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4951.416504] LNetError: 324108:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4951.426588] LNet: Removed LNI 192.168.201.158@tcp [ 4951.919201] Key type .llcrypt unregistered [ 4951.921489] Key type ._llcrypt unregistered [ 4965.361578] Key type ._llcrypt registered [ 4965.363772] Key type .llcrypt registered [ 4965.427205] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4965.854472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4965.864774] alg: No test for adler32 (adler32-zlib) [ 4966.739762] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4966.834876] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4968.432197] Key type lgssc registered [ 4968.996885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4972.904574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4972.918650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.123114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4975.698462] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4979.843799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4980.037190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4981.923089] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4984.742488] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4986.626255] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4989.997512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4990.126810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.203603] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4994.924560] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5004.774636] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5004.774924] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5004.777284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5004.789886] Lustre: Skipped 1 previous similar message [ 5005.410564] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 5005.414808] Lustre: Skipped 1 previous similar message [ 5009.484857] LustreError: 327903:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5009.513792] Lustre: server umount lustre-MDT0001 complete [ 5009.893363] LustreError: 326243:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.902928] LustreError: 326243:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5010.517074] LustreError: 326243:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5010.935643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5014.796369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5014.854763] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5014.857973] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5014.880436] Lustre: 325614:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5015.043754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5017.034225] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5020.140792] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5025.264756] LustreError: 329078:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5025.267054] LustreError: 329078:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5025.269353] LustreError: 325617:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5025.286577] LustreError: 329081:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.201.158@tcp: cfg command failed: rc = -17 [ 5025.289206] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5025.289206] [ 5322.213124] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.215274] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5322.226186] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5322.233144] Lustre: Skipped 1 previous similar message [ 5326.150501] LustreError: 339569:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5326.153761] LustreError: 339569:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5326.185472] Lustre: server umount lustre-MDT0001 complete [ 5327.330269] LustreError: 325625:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.339542] LustreError: 325625:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5327.367216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5330.301991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5330.335414] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5330.336996] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5330.348600] Lustre: 325612:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5330.488605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5331.820646] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5335.533062] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5336.556904] LustreError: 340640:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5336.561569] LustreError: 340640:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5336.566829] LustreError: 325617:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5336.571971] LustreError: 325617:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5336.599786] LustreError: 340643:0:(obd_config.c:685:class_setup()) Device 8 already setup (type osp) [ 5336.602483] LustreError: 340643:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.201.158@tcp: cfg command failed: rc = -17 [ 5336.605445] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.158@tcp [ 5336.605445] [ 5635.163088] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 5635.167166] Lustre: Skipped 1 previous similar message [ 5635.557561] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.564139] Lustre: Skipped 1 previous similar message [ 5640.275694] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 5640.280128] Lustre: Skipped 2 previous similar messages [ 5648.352229] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5648.398301] LustreError: 351478:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5648.400298] LustreError: 351478:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5648.429746] Lustre: server umount lustre-MDT0001 complete [ 5649.560416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5650.515237] LustreError: 326243:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5650.519362] LustreError: 326243:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5652.867227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5652.909370] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5652.912068] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5652.931668] Lustre: 325614:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5653.071668] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5654.636443] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5658.095352] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5660.135460] LustreError: 352558:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5660.138488] LustreError: 352558:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5660.141936] LustreError: 325617:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5660.144363] LustreError: 325617:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5660.169600] LustreError: 352565:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 5660.177509] LustreError: 352565:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.201.158@tcp: cfg command failed: rc = -114 [ 5660.182454] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 5660.182454] [ 5960.161347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5960.164279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.168363] Lustre: Skipped 1 previous similar message [ 5960.170307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5960.172294] Lustre: Skipped 5 previous similar messages [ 5965.010350] LustreError: 363485:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5965.013853] LustreError: 363485:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5965.046792] Lustre: server umount lustre-MDT0000 complete [ 5966.698962] LustreError: 325605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701389 with bad export cookie 3334363732718739690 [ 5966.706179] LustreError: 325605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5966.708264] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5968.353618] LustreError: 325621:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.354153] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5968.363543] LustreError: 325621:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5968.371069] Lustre: Skipped 2 previous similar messages [ 5972.876925] LustreError: 363686:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5972.881890] LustreError: 363686:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5972.987030] Lustre: server umount lustre-MDT0001 complete [ 5974.753534] Lustre: server umount lustre-OST0000 complete [ 5978.478553] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 5982.282934] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5986.505130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5989.962492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5992.499204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5995.163367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5997.826676] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 10:50:19 (1755701419) [ 6000.662382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6000.686839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.759916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6000.769922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6000.802986] Lustre: lustre-MDT0000: new disk, initializing [ 6000.825054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6000.830721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6002.080695] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6007.149603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6007.186694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6007.245604] Lustre: 366409:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6007.278955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6007.282693] Lustre: Skipped 1 previous similar message [ 6007.331542] Lustre: lustre-MDT0001: new disk, initializing [ 6007.385926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.406395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6007.412328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6008.977648] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6012.656031] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6013.967341] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6017.082642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6017.120171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6017.270050] Lustre: lustre-OST0000: new disk, initializing [ 6017.272259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6017.305798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.869048] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6023.684811] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6024.200094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6024.205783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6024.261448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6032.865411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6032.870087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.878161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.881866] Lustre: Skipped 5 previous similar messages [ 6038.093045] LustreError: 368832:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6038.096911] LustreError: 368832:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6038.133686] Lustre: server umount lustre-MDT0000 complete [ 6039.523632] LustreError: 366418:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.532862] LustreError: 366418:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6039.717099] LustreError: 367181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701462 with bad export cookie 3334363732718741643 [ 6039.721561] LustreError: 367193:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 6039.723753] LustreError: 367181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6039.723832] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6039.907387] Lustre: server umount lustre-MDT0001 complete [ 6051.905018] Lustre: server umount lustre-OST0000 complete [ 6064.825115] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 6069.256022] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6074.210940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6077.329242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6080.326498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6083.390064] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6084.218506] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5958 sec ========== 10:51:45 (1755701505) [ 6084.900557] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:51:46 (1755701506) === [ 6086.253270] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:51:47 (1755701507) === [ 6103.949946] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 6105.329809] Key type lgssc unregistered [ 6105.487604] LNet: 374320:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6105.492402] LNetError: 374320:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6105.506679] LNet: Removed LNI 192.168.201.158@tcp [ 6105.917574] Key type .llcrypt unregistered [ 6105.919675] Key type ._llcrypt unregistered