[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 556641326 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003105] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010151] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.013073] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.022678] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027441] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] *** VALIDATE tmpfs *** [ 0.032441] *** VALIDATE proc *** [ 0.034130] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037037] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041029] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.044837] debug: unmapping init [mem 0xffffffff9e059000-0xffffffff9e060fff] [ 0.046171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048031] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055390] rcu: Hierarchical SRCU implementation. [ 0.057449] smp: Bringing up secondary CPUs ... [ 0.058648] x86: Booting SMP configuration: [ 0.059029] .... node #0, CPUs: #1 #2 #3 [ 0.062369] smp: Brought up 1 node, 4 CPUs [ 0.064012] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.211771] node 0 deferred pages initialised in 143ms [ 0.216193] devtmpfs: initialized [ 0.217000] x86/mm: Memory block size: 128MB [ 0.220906] gcov: version magic: 0x41383552 [ 0.224327] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.228117] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.231373] pinctrl core: initialized pinctrl subsystem [ 0.233264] [ 0.233815] ************************************************************* [ 0.236018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239018] ** ** [ 0.241013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244015] ** ** [ 0.246013] ** This means that this kernel is built to expose internal ** [ 0.248017] ** IOMMU data structures, which may compromise security on ** [ 0.251014] ** your system. ** [ 0.253011] ** ** [ 0.255013] ** If you see this message and you are not debugging the ** [ 0.257014] ** kernel, report this immediately to your vendor! ** [ 0.259015] ** ** [ 0.261014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264018] ************************************************************* [ 0.266725] NET: Registered protocol family 16 [ 0.268493] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.271068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.274073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277139] cpuidle: using governor menu [ 0.278677] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281501] PCI: Using configuration type 1 for base access [ 0.283117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.292150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.293000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295645] cryptd: max_cpu_qlen set to 1000 [ 0.298137] ACPI: Added _OSI(Module Device) [ 0.299013] ACPI: Added _OSI(Processor Device) [ 0.300009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.302017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.305774] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.312345] ACPI: Interpreter enabled [ 0.314068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.316015] ACPI: Using IOAPIC for interrupt routing [ 0.317095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.321418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.330087] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.332038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.335020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.338106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.344275] acpiphp: Slot [2] registered [ 0.345136] acpiphp: Slot [3] registered [ 0.347108] acpiphp: Slot [4] registered [ 0.348118] acpiphp: Slot [5] registered [ 0.350127] acpiphp: Slot [6] registered [ 0.351102] acpiphp: Slot [7] registered [ 0.352131] acpiphp: Slot [8] registered [ 0.354121] acpiphp: Slot [9] registered [ 0.355126] acpiphp: Slot [10] registered [ 0.357114] acpiphp: Slot [11] registered [ 0.358098] acpiphp: Slot [12] registered [ 0.360106] acpiphp: Slot [13] registered [ 0.362131] acpiphp: Slot [14] registered [ 0.363079] acpiphp: Slot [15] registered [ 0.364087] acpiphp: Slot [16] registered [ 0.365101] acpiphp: Slot [17] registered [ 0.367080] acpiphp: Slot [18] registered [ 0.368126] acpiphp: Slot [19] registered [ 0.369085] acpiphp: Slot [20] registered [ 0.371107] acpiphp: Slot [21] registered [ 0.373108] acpiphp: Slot [22] registered [ 0.377187] acpiphp: Slot [23] registered [ 0.379110] acpiphp: Slot [24] registered [ 0.381125] acpiphp: Slot [25] registered [ 0.382095] acpiphp: Slot [26] registered [ 0.383095] acpiphp: Slot [27] registered [ 0.385105] acpiphp: Slot [28] registered [ 0.388132] acpiphp: Slot [29] registered [ 0.389129] acpiphp: Slot [30] registered [ 0.391126] acpiphp: Slot [31] registered [ 0.392070] PCI host bridge to bus 0000:00 [ 0.394020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.395019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.397021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.399019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.401030] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.404025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.406163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.410481] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.414560] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.426020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.431028] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.433020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.436022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.440030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.442452] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.445788] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.448054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.451727] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.457018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.473032] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.477018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.482231] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.488016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.495014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.510028] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.520042] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.526018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.532019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.547025] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.558055] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.564013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.570016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.581014] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.591171] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.597013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.603017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.619026] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.629953] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.634018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.641031] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.654018] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.661972] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.668016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.675020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.692015] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.704459] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.706355] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.709384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.711373] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.713244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.718036] iommu: Default domain type: Passthrough [ 0.720429] SCSI subsystem initialized [ 0.722130] ACPI: bus type USB registered [ 0.723117] usbcore: registered new interface driver usbfs [ 0.725060] usbcore: registered new interface driver hub [ 0.726049] usbcore: registered new device driver usb [ 0.728197] pps_core: LinuxPPS API ver. 1 registered [ 0.730013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.734101] PTP clock support registered [ 0.737111] EDAC MC: Ver: 3.0.0 [ 0.739149] PCI: Using ACPI for IRQ routing [ 0.741620] NetLabel: Initializing [ 0.742000] NetLabel: domain hash size = 128 [ 0.744018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.746110] NetLabel: unlabeled traffic allowed by default [ 0.749289] vgaarb: loaded [ 0.750197] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.761784] clocksource: Switched to clocksource kvm-clock [ 0.871221] VFS: Disk quotas dquot_6.6.0 [ 0.872914] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.876035] *** VALIDATE ramfs *** [ 0.876955] *** VALIDATE hugetlbfs *** [ 0.878339] pnp: PnP ACPI init [ 0.880338] pnp: PnP ACPI: found 6 devices [ 0.897123] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.899701] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.902053] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.904196] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.906457] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.908782] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.911342] NET: Registered protocol family 2 [ 0.913675] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917980] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921472] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.926969] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.929923] TCP: Hash tables configured (established 65536 bind 65536) [ 0.932727] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.935744] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.938639] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942960] NET: Registered protocol family 1 [ 0.945483] RPC: Registered named UNIX socket transport module. [ 0.947583] RPC: Registered udp transport module. [ 0.948927] RPC: Registered tcp transport module. [ 0.950521] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.952652] NET: Registered protocol family 44 [ 0.954368] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.956447] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.959032] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.961383] PCI: CLS 0 bytes, default 64 [ 0.963198] Unpacking initramfs... [ 2.379244] debug: unmapping init [mem 0xffff9519bcc54000-0xffff9519bffbffff] [ 2.382815] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.384811] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.387288] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.891063] Initialise system trusted keyrings [ 2.892928] Key type blacklist registered [ 2.895345] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.905425] zbud: loaded [ 2.908379] *** VALIDATE nfs *** [ 2.909814] *** VALIDATE nfs4 *** [ 2.912660] pstore: using deflate compression [ 2.916676] Platform Keyring initialized [ 3.051724] NET: Registered protocol family 38 [ 3.053803] Key type asymmetric registered [ 3.055320] Asymmetric key parser 'x509' registered [ 3.057198] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.060534] io scheduler mq-deadline registered [ 3.062132] io scheduler kyber registered [ 3.063817] io scheduler bfq registered [ 3.065866] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.069160] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.072148] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.075211] ACPI: Power Button [PWRF] [ 3.174542] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.278485] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.478592] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.578062] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.776877] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.806547] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.840366] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.849161] Non-volatile memory driver v1.3 [ 3.850747] Linux agpgart interface v0.103 [ 3.884875] virtio_blk virtio1: [vda] 133280 512-byte logical blocks (68.2 MB/65.1 MiB) [ 3.888298] vda: detected capacity change from 0 to 68239360 [ 3.902384] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.905320] vdb: detected capacity change from 0 to 1073741824 [ 3.923083] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.928896] vdc: detected capacity change from 0 to 2621440000 [ 3.944437] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.948872] vdd: detected capacity change from 0 to 2621440000 [ 3.963073] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.965940] vde: detected capacity change from 0 to 4294967296 [ 3.981872] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.984238] vdf: detected capacity change from 0 to 4294967296 [ 3.989558] libphy: Fixed MDIO Bus: probed [ 3.994893] usbcore: registered new interface driver usbserial_generic [ 3.997530] usbserial: USB Serial support registered for generic [ 4.000145] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.005349] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.007316] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.009747] mousedev: PS/2 mouse device common for all mice [ 4.014933] rtc_cmos 00:05: RTC can wake from S4 [ 4.017708] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.020937] rtc_cmos 00:05: registered as rtc0 [ 4.022014] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.023767] intel_pstate: CPU model not supported [ 4.029039] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.032200] hid: raw HID events driver (C) Jiri Kosina [ 4.037801] usbcore: registered new interface driver usbhid [ 4.040562] usbhid: USB HID core driver [ 4.043464] drop_monitor: Initializing network drop monitor service [ 4.043821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.045855] Initializing XFRM netlink socket [ 4.051859] NET: Registered protocol family 10 [ 4.054809] Segment Routing with IPv6 [ 4.056193] NET: Registered protocol family 17 [ 4.058304] mpls_gso: MPLS GSO support [ 4.063875] RAS: Correctable Errors collector initialized. [ 4.066261] AVX version of gcm_enc/dec engaged. [ 4.068254] AES CTR mode by8 optimization enabled [ 4.158174] sched_clock: Marking stable (4158017492, 0)->(5084381062, -926363570) [ 4.162571] registered taskstats version 1 [ 4.165033] Loading compiled-in X.509 certificates [ 4.168833] zswap: loaded using pool lzo/zbud [ 4.193844] Key type big_key registered [ 4.230657] Key type encrypted registered [ 4.232323] ima: No TPM chip found, activating TPM-bypass! [ 4.234609] ima: Allocated hash algorithm: sha1 [ 4.236504] ima: No architecture policies found [ 4.238635] evm: Initialising EVM extended attributes: [ 4.240790] evm: security.selinux [ 4.241793] evm: security.ima [ 4.243070] evm: security.capability [ 4.244265] evm: HMAC attrs: 0x1 [ 4.247808] rtc_cmos 00:05: setting system clock to 2025-11-07 04:26:19 UTC (1762489579) [ 4.254320] debug: unmapping init [mem 0xffffffff9f003000-0xffffffff9f1fffff] [ 4.257336] debug: unmapping init [mem 0xffffffff9dd82000-0xffffffff9e058fff] [ 4.263179] Write protecting the kernel read-only data: 28672k [ 4.266756] debug: unmapping init [mem 0xffffffff9c403000-0xffffffff9c5fffff] [ 4.269622] debug: unmapping init [mem 0xffffffff9cd14000-0xffffffff9cdfffff] [ 4.302886] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.310838] systemd[1]: Detected virtualization kvm. [ 4.313323] systemd[1]: Detected architecture x86-64. [ 4.315320] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.343394] systemd[1]: No hostname configured. [ 4.344817] systemd[1]: Set hostname to . [ 4.346542] random: systemd: uninitialized urandom read (16 bytes read) [ 4.348643] systemd[1]: Initializing machine ID from random generator. [ 4.387035] random: ln: uninitialized urandom read (6 bytes read) [ 4.490920] random: systemd: uninitialized urandom read (16 bytes read) [ 4.493417] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.497542] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.501531] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.128665] device-mapper: uevent: version 1.0.3 [ 5.131203] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] [ 5.850217] random: fast init done Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.912990] virtio_net virtio0 ens2: renamed from eth0 [ 5.961396] scsi host0: ata_piix [ 5.964500] scsi host1: ata_piix [ 5.965623] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.973202] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.729534] dracut-initqueue[596]: RTNETLINK answers: File exists [ 10.749936] random: crng init done [ 10.751253] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.253538] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.450481] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.739913] SELinux: Disabled at runtime. [ 12.807610] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.816472] systemd[1]: Detected virtualization kvm. [ 12.818171] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.360157] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.364076] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.368863] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.373505] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.377904] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.391612] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.399696] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ 13.501851] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.858161] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.126439] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.218135] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.327421] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.345918] EDAC sbridge: Ver: 1.1.2 [ 16.053152] Key type dns_resolver registered [ 16.354289] NFS: Registering the id_resolver key type [ 16.356087] Key type id_resolver registered [ 16.357312] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg403-server login: [ 54.096025] hrtimer: interrupt took 3997822 ns [ 68.129740] libcfs: loading out-of-tree module taints kernel. [ 68.157849] Key type ._llcrypt registered [ 68.160700] Key type .llcrypt registered [ 68.307586] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 85.179971] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 86.321220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.346366] alg: No test for adler32 (adler32-zlib) [ 87.707555] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 88.292368] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 90.071175] Key type lgssc registered [ 91.569531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.290893] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 109.897311] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.413501] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.714578] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.955524] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.304741] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 156.682933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.743984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.768979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.062460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.085032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.207191] Lustre: lustre-MDT0000: new disk, initializing [ 158.326821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.369728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.159502] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.329925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.444433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 175.560897] Lustre: 6528:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 175.619315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 175.628045] Lustre: Skipped 1 previous similar message [ 175.718959] Lustre: lustre-MDT0001: new disk, initializing [ 175.812453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 175.850165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 175.854249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 179.432454] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.366833] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 195.605562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.691388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 195.851982] Lustre: lustre-OST0000: new disk, initializing [ 195.857647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 195.914611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.254670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.262354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.355490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.767669] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.126279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.226855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 215.440979] Lustre: lustre-OST0001: new disk, initializing [ 215.443786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 215.513642] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.448379] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.724821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 222.729335] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 222.792509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 232.424636] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.314593] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.121790] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.165971] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing yml_node [ 258.676219] Lustre: DEBUG MARKER: Client: 2.16.58.106 [ 260.929704] Lustre: DEBUG MARKER: MDS: 2.16.58.106 [ 263.258811] Lustre: DEBUG MARKER: OSS: 2.16.58.106 [ 264.754824] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Nov 6 23:30:39 EST 2025 [ 280.006333] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 281.164996] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.977181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 298.988153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.010721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.492072] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.508038] Lustre: Skipped 2 previous similar messages [ 299.516276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.526112] Lustre: Skipped 3 previous similar messages [ 301.374312] LustreError: 12231:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 301.416919] Lustre: server umount lustre-MDT0000 complete [ 305.122056] LustreError: 6521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762489880 with bad export cookie 3599062875374985098 [ 305.123434] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.134935] LustreError: 6521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 309.728290] LustreError: 6536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.729363] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.730177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.739436] LustreError: 6536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 309.760480] Lustre: Skipped 1 previous similar message [ 310.752810] LustreError: 6534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.783143] LustreError: 6534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 311.396484] LustreError: 12432:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 311.399361] LustreError: 12432:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 311.562325] Lustre: server umount lustre-MDT0001 complete [ 314.837454] LustreError: 12634:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 314.842710] LustreError: 12634:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 314.879596] Lustre: server umount lustre-OST0000 complete [ 318.396921] LustreError: 12834:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 318.403528] LustreError: 12834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 318.570355] Lustre: server umount lustre-OST0001 complete [ 324.971612] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 333.745900] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 342.451747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.305738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 355.948280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 362.043218] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 368.528319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 368.594642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.820469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 368.856077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 368.932347] Lustre: lustre-MDT0000: new disk, initializing [ 369.049808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.071253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 373.272972] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 382.721550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 382.769084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 382.850392] Lustre: 15021:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 382.906283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 382.909268] Lustre: Skipped 1 previous similar message [ 383.009520] Lustre: lustre-MDT0001: new disk, initializing [ 383.132646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 383.175699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 383.190491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 387.136642] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 393.898769] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 396.946328] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 402.988810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 403.083398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 403.364902] Lustre: lustre-OST0000: new disk, initializing [ 403.368089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 403.473341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.550400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 404.563872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.270512] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 410.114407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.479694] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 419.098716] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 419.271789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 423.530656] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 423.815417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 430.565863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 430.569785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.571037] LustreError: Skipped 1 previous similar message [ 430.578049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.585780] Lustre: Skipped 1 previous similar message [ 430.602725] Lustre: Skipped 4 previous similar messages [ 435.681800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.690750] Lustre: Skipped 1 previous similar message [ 440.287637] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 440.331092] LustreError: 17960:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 440.337329] LustreError: 17960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 440.529405] Lustre: server umount lustre-OST0000 complete [ 444.897316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 444.900296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.922802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.509563] LustreError: 18164:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 449.529891] LustreError: 18164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 449.670119] Lustre: server umount lustre-MDT0000 complete [ 451.040612] LustreError: 16779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.057455] LustreError: 16779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 452.725134] LustreError: 15013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490027 with bad export cookie 3599062875374986631 [ 452.733494] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 453.105864] Lustre: server umount lustre-MDT0001 complete [ 457.602128] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 23:33:51 (1762490031) [ 463.343367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.952747] LustreError: 18889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.025561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.567561] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 469.480927] LustreError: 18888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.501789] LustreError: 18888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 475.799292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 479.552460] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 485.585584] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 489.365657] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 496.362341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 503.255405] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 509.102613] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 518.327568] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 525.794812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.799860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 525.822444] Lustre: Skipped 3 previous similar messages [ 525.823258] Lustre: Skipped 3 previous similar messages [ 530.335304] LustreError: 21191:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 530.343735] LustreError: 21191:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 530.433114] Lustre: server umount lustre-MDT0000 complete [ 530.934868] LustreError: 19613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.952197] LustreError: 19613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 533.817910] LustreError: 18870:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490109 with bad export cookie 3599062875374987513 [ 533.825212] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.831935] LustreError: 18870:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 534.226724] Lustre: server umount lustre-MDT0001 complete [ 548.577809] Lustre: server umount lustre-OST0000 complete [ 568.799586] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 577.537891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.809613] LustreError: 22921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.863522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.866807] Lustre: Skipped 2 previous similar messages [ 581.211961] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 588.476308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 592.314291] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 595.262223] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 602.165736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 608.224187] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 616.087397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 616.159945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 616.312787] Lustre: lustre-OST0001: new disk, initializing [ 616.317962] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 618.314100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 618.336040] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 618.445804] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 621.570428] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 630.029823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 633.037303] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 650.209487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 650.212702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.221380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.223883] Lustre: Skipped 1 previous similar message [ 655.816159] LustreError: 26399:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 655.818899] LustreError: 26399:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 655.936537] Lustre: server umount lustre-MDT0000 complete [ 659.427139] LustreError: 23954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.444315] LustreError: 23954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 659.753436] LustreError: 22901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490235 with bad export cookie 3599062875374988353 [ 659.761331] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.779737] LustreError: 22901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 660.298602] Lustre: server umount lustre-MDT0001 complete [ 673.827743] Lustre: server umount lustre-OST0000 complete [ 687.649617] Lustre: server umount lustre-OST0001 complete [ 702.375708] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 710.990338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.612177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.617586] Lustre: Skipped 3 previous similar messages [ 715.749210] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 724.966593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 730.115105] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 733.552470] Lustre: 29298:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 740.138928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 746.681082] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 749.771831] LustreError: 29654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.803940] LustreError: 29654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 756.058036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 762.483216] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 770.383542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 774.169312] Lustre: 31157:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 786.915176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 786.917973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.931034] Lustre: Skipped 3 previous similar messages [ 786.947788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.953351] Lustre: Skipped 4 previous similar messages [ 790.757900] LustreError: 31358:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 790.762909] LustreError: 31358:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 790.968091] Lustre: server umount lustre-MDT0000 complete [ 794.993551] LustreError: 28159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490370 with bad export cookie 3599062875374989655 [ 794.996577] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 795.005992] LustreError: 28159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 795.472526] Lustre: server umount lustre-MDT0001 complete [ 809.895034] Lustre: server umount lustre-OST0000 complete [ 814.047135] Lustre: 3672:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762490373/real 1762490373] req@ffff951a06dfa680 x1848104361155072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762490389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 814.495977] Lustre: server umount lustre-OST0001 complete [ 825.092380] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 23:39:59 (1762490399) [ 856.931317] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 868.141850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.626730] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 878.060488] LustreError: 33942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.101646] LustreError: 33942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 882.834719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 887.914600] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 891.581436] Lustre: 35058:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 899.222359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 905.863403] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 915.137561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 921.793422] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 930.256084] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 934.332241] Lustre: 36915:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 946.144944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.152679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.185758] Lustre: Skipped 6 previous similar messages [ 946.210195] Lustre: Skipped 7 previous similar messages [ 951.377570] LustreError: 37070:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 951.396114] LustreError: 37070:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 951.536808] Lustre: server umount lustre-MDT0000 complete [ 955.573252] LustreError: 33922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490530 with bad export cookie 3599062875374990761 [ 955.582622] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 955.593796] LustreError: 33922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 956.328195] Lustre: server umount lustre-MDT0001 complete [ 970.772533] Lustre: server umount lustre-OST0000 complete [ 972.713000] Lustre: 3670:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762490531/real 1762490531] req@ffff951a3d67a680 x1848104361208192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762490547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 974.829978] Lustre: server umount lustre-OST0001 complete [ 984.349580] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 23:42:38 (1762490558) [ 1000.846310] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1011.337418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.877773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.890467] Lustre: Skipped 7 previous similar messages [ 1015.553739] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1024.177677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1028.295479] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1031.405400] Lustre: 40162:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1037.954175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1043.438022] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1052.124403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1058.647791] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1068.200654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1073.183661] Lustre: 42018:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1079.307539] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1079.310826] Lustre: Skipped 1 previous similar message [ 1083.361955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.378304] Lustre: Skipped 4 previous similar messages [ 1088.389189] Lustre: server umount lustre-MDT0000 complete [ 1092.344100] LustreError: 39025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490667 with bad export cookie 3599062875374991867 [ 1092.351932] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1098.811533] Lustre: server umount lustre-MDT0001 complete [ 1106.819478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.181391] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x31f2709dd305bea1 [ 1118.212881] Lustre: MGC192.168.204.103@tcp: Connection restored to 0@lo (at 0@lo) [ 1122.736356] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1127.919631] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1131.099862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1135.774264] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1137.141710] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1137.145690] Lustre: Skipped 1 previous similar message [ 1141.778332] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.659251] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 1149.000834] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1151.257764] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1157.088228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1163.566637] Lustre: server umount lustre-MDT0000 complete [ 1167.843500] LustreError: 42896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.883408] LustreError: 42896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 1168.388860] LustreError: 39027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490743 with bad export cookie 3599062875374993057 [ 1168.396975] LustreError: 39027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1168.401669] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.011440] Lustre: server umount lustre-MDT0001 complete [ 1184.349343] Lustre: server umount lustre-OST0000 complete [ 1199.077725] Lustre: server umount lustre-OST0001 complete [ 1210.455797] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 23:46:24 (1762490784) [ 1228.083582] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1240.938590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.922351] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1256.913192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1262.049927] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1265.537561] Lustre: 47513:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1273.104189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1279.289582] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1288.551493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1295.385884] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1303.612281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1313.767401] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1326.977771] LustreError: 49595:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1326.998899] LustreError: 49595:0:(obd_class.h:479:obd_check_dev()) Skipped 67 previous similar messages [ 1327.313927] Lustre: server umount lustre-MDT0000 complete [ 1329.120541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1330.961378] LustreError: 49432:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490906 with bad export cookie 3599062875374993841 [ 1330.964982] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1330.966488] LustreError: 49432:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1331.217829] Lustre: server umount lustre-MDT0001 complete [ 1345.712210] Lustre: server umount lustre-OST0000 complete [ 1360.405201] Lustre: server umount lustre-OST0001 complete [ 1370.082449] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 23:49:04 (1762490944) [ 1372.555838] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1374.171068] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 23:49:08 (1762490948) [ 1384.599815] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 1394.027986] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1404.094318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1410.054937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1417.131736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1417.219361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.416820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1417.510690] Lustre: lustre-MDT0000: new disk, initializing [ 1417.691918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1421.687437] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1429.174142] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1435.557421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1435.640791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1435.851745] Lustre: lustre-OST0000: new disk, initializing [ 1435.861476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1435.864242] Lustre: Skipped 1 previous similar message [ 1437.596888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1437.603826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1437.646757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1440.956881] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1448.446354] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1450.469328] Lustre: DEBUG MARKER: create test files [ 1478.627085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1478.631888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.642619] Lustre: Skipped 11 previous similar messages [ 1478.649634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1478.652556] Lustre: Skipped 22 previous similar messages [ 1484.714440] Lustre: server umount lustre-OST0000 complete [ 1487.975893] Lustre: server umount lustre-MDT0000 complete [ 1498.608929] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 1501.566554] Key type lgssc unregistered [ 1501.884305] LNet: 54933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.897335] LNetError: 54933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.918782] LNet: Removed LNI 192.168.204.103@tcp [ 1502.873170] Key type .llcrypt unregistered [ 1502.877625] Key type ._llcrypt unregistered [ 1526.832330] Key type ._llcrypt registered [ 1526.841745] Key type .llcrypt registered [ 1526.944730] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1527.755766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1527.820267] alg: No test for adler32 (adler32-zlib) [ 1528.918063] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1529.142982] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 1530.824409] Key type lgssc registered [ 1532.132878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1539.851717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1539.882584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.261556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.807469] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1550.432543] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1557.236440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1557.563103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.382910] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1566.698333] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1566.766296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1566.864478] Lustre: 57519:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1566.872612] Lustre: 57519:0:(osd_handler.c:2078:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1566.884163] Lustre: 57519:0:(osd_handler.c:2085:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1566.890098] Lustre: 57519:0:(osd_handler.c:2095:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1566.894309] Lustre: 57519:0:(osd_handler.c:2102:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1566.899298] Lustre: 57519:0:(osd_handler.c:2109:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1571.192790] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1575.749274] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1582.048900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.052805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1582.070834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1586.549288] LustreError: 58264:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1586.735115] Lustre: server umount lustre-OST0000 complete [ 1590.127614] LustreError: 58466:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1590.136858] LustreError: 58466:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1590.392764] Lustre: server umount lustre-MDT0000 complete [ 1599.299609] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 1601.660488] Key type lgssc unregistered [ 1601.938711] LNet: 58996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.949881] LNetError: 58996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.964553] LNet: Removed LNI 192.168.204.103@tcp [ 1602.755649] Key type .llcrypt unregistered [ 1602.758303] Key type ._llcrypt unregistered [ 1631.737266] Key type ._llcrypt registered [ 1631.739116] Key type .llcrypt registered [ 1631.844933] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1632.841852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1632.898617] alg: No test for adler32 (adler32-zlib) [ 1633.867798] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1634.055159] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 1635.703255] Key type lgssc registered [ 1636.621940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1643.385334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1643.420345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.839134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.496644] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1651.559569] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.510254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1657.921482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.322497] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1669.150865] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1670.201326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1670.262965] Lustre: 61704:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1670.267155] Lustre: 61704:0:(osd_handler.c:2078:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1670.272697] Lustre: 61704:0:(osd_handler.c:2085:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1670.280125] Lustre: 61704:0:(osd_handler.c:2095:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1670.292373] Lustre: 61704:0:(osd_handler.c:2102:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1670.298449] Lustre: 61704:0:(osd_handler.c:2109:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1680.492594] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1690.593324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.595879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.616517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.137093] LustreError: 62417:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1694.411828] Lustre: server umount lustre-OST0000 complete [ 1698.366908] LustreError: 62619:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1698.369941] LustreError: 62619:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1698.654528] Lustre: server umount lustre-MDT0000 complete [ 1708.336778] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 1711.723368] Key type lgssc unregistered [ 1712.049590] LNet: 63152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.057429] LNetError: 63152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.071860] LNet: Removed LNI 192.168.204.103@tcp [ 1712.786693] Key type .llcrypt unregistered [ 1712.788263] Key type ._llcrypt unregistered [ 1728.704833] Key type ._llcrypt registered [ 1728.707571] Key type .llcrypt registered [ 1728.815430] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 1743.736280] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1745.042929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.096712] alg: No test for adler32 (adler32-zlib) [ 1746.127084] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1746.366552] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 1748.031219] Key type lgssc registered [ 1749.295159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1759.713511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1766.538739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1772.893659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1779.046770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1784.844110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1784.882709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1784.895044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.066745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1786.096622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1786.164766] Lustre: lustre-MDT0000: new disk, initializing [ 1786.255444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.274440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1789.658524] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1798.166593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1798.224251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1798.298688] Lustre: 66282:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1798.359477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1798.362585] Lustre: Skipped 1 previous similar message [ 1798.424434] Lustre: lustre-MDT0001: new disk, initializing [ 1798.486574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1798.535805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1798.553928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1801.127550] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1807.424563] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.525697] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1816.233677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1816.293620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1816.452609] Lustre: lustre-OST0000: new disk, initializing [ 1816.456862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1816.515108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.841972] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1822.257975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1822.296660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1822.408809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1827.971852] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1831.761310] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1831.917873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1835.497549] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1835.668984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1838.054416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1838.056422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.061691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1838.076745] LustreError: Skipped 1 previous similar message [ 1838.104799] Lustre: Skipped 1 previous similar message [ 1843.173645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1843.192226] Lustre: Skipped 2 previous similar messages [ 1843.456456] LustreError: 69221:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1843.692164] Lustre: server umount lustre-OST0000 complete [ 1848.289350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.291250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.311626] Lustre: Skipped 1 previous similar message [ 1853.204727] LustreError: 69422:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1853.219293] LustreError: 69422:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1853.325815] Lustre: server umount lustre-MDT0000 complete [ 1856.788781] LustreError: 66272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491432 with bad export cookie 4269339896587559076 [ 1856.795473] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1856.802048] LustreError: 66272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1856.912689] LustreError: 69626:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1856.923410] LustreError: 69626:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1857.234202] Lustre: server umount lustre-MDT0001 complete [ 1864.808668] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 23:57:19 (1762491439) [ 1873.883692] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1886.010986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1909.614725] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 1920.696849] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1930.991633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1936.792678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1942.356048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1948.353290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1954.768487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1954.821908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.010978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1955.041566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1955.126653] Lustre: lustre-MDT0000: new disk, initializing [ 1955.196590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.215396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1959.146231] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1969.241588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1969.295211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1969.389326] Lustre: 73864:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1969.446111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1969.451276] Lustre: Skipped 1 previous similar message [ 1969.547189] Lustre: lustre-MDT0001: new disk, initializing [ 1969.655926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.691923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1969.702968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1973.340822] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1979.539552] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1982.249840] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1987.577304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1987.631820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1987.862054] Lustre: lustre-OST0000: new disk, initializing [ 1987.869302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1987.924275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.003383] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1993.244081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1993.248795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1993.326341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1997.726669] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2000.874714] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2001.027375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2003.917505] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2004.122686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2008.551771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.553195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2008.562469] Lustre: Skipped 1 previous similar message [ 2008.574449] Lustre: Skipped 1 previous similar message [ 2008.577421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.589398] LustreError: 76804:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2011.591481] LustreError: 76804:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2011.739813] Lustre: server umount lustre-OST0000 complete [ 2015.714177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2015.718927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.729660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.907534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.914398] Lustre: Skipped 3 previous similar messages [ 2029.023286] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2029.122739] LustreError: 77006:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2029.128240] LustreError: 77006:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2029.292311] Lustre: server umount lustre-MDT0000 complete [ 2032.628393] LustreError: 73858:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491607 with bad export cookie 4269339896587559958 [ 2032.629536] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2032.662966] LustreError: 73858:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2033.011114] Lustre: server umount lustre-MDT0001 complete [ 2042.060327] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 00:00:15 (1762491615) [ 2048.255859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.660494] LustreError: 77884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.688516] LustreError: 77884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2048.731046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.664723] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2054.117478] LustreError: 77884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.210483] LustreError: 77884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.276256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2063.030932] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2068.992175] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2072.994013] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2079.016284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2084.264534] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2089.246713] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2092.469584] Lustre: 80234:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting e872ca27-998d-4033-97df-ea91620a621f at adminstrative request [ 2093.666543] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2093.696463] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2103.273439] Lustre: MGS: Client e872ca27-998d-4033-97df-ea91620a621f (at 0@lo) reconnecting [ 2103.285535] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x3b3fbe041981277a to 0x3b3fbe0419812a1a [ 2103.304139] Lustre: MGC192.168.204.103@tcp: Connection restored to 0@lo (at 0@lo) [ 2128.315069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2128.387458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2128.515328] Lustre: lustre-OST0001: new disk, initializing [ 2128.521867] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2128.613423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2128.618241] Lustre: Skipped 2 previous similar messages [ 2133.314780] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2134.071159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2134.082360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2134.171300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2138.977762] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2144.741906] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2144.743384] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.746569] LustreError: Skipped 1 previous similar message [ 2144.756568] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2144.760546] Lustre: Skipped 2 previous similar messages [ 2144.775600] Lustre: Skipped 3 previous similar messages [ 2147.121043] LustreError: 81369:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2147.128699] LustreError: 81369:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2147.173682] Lustre: server umount lustre-OST0001 complete [ 2149.861156] LustreError: 79484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.874520] LustreError: 79484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2154.982205] LustreError: 81009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.993384] LustreError: 81009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2155.683055] Lustre: server umount lustre-OST0000 complete [ 2160.096413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.110374] Lustre: Skipped 3 previous similar messages [ 2164.474282] LustreError: 81774:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2164.480302] LustreError: 81774:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2164.552137] Lustre: server umount lustre-MDT0000 complete [ 2165.216221] LustreError: 77883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.231876] LustreError: 77883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2167.303438] LustreError: 77864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491742 with bad export cookie 4269339896587561498 [ 2167.308304] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.313068] LustreError: 77864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2167.603089] Lustre: server umount lustre-MDT0001 complete [ 2175.386277] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 00:02:29 (1762491749) [ 2176.517400] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2178.065282] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 00:02:32 (1762491752) [ 2179.571184] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2181.290840] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 00:02:35 (1762491755) [ 2182.810966] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2184.391754] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 00:02:38 (1762491758) [ 2185.668509] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2189.962635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2190.838293] loop: module loaded [ 2190.869719] loop0: detected capacity change from 0 to 4294967296 [ 2190.917798] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2192.422507] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2195.017109] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2198.390370] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2206.425712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2209.982271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2213.976798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2217.851933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2223.964817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2224.013416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2224.038164] LustreError: 84550:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2224.050489] LustreError: 84550:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2224.056286] LustreError: 84550:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2224.064635] LustreError: 84550:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2238.223989] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 2245.200137] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 2252.097297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2258.360752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2264.791243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2271.394038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2277.948626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2278.001948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.278838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2278.348572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2278.461986] Lustre: lustre-MDT0000: new disk, initializing [ 2278.543547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.570476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2281.545436] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2290.075330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2290.128443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2290.191667] Lustre: 87395:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2290.397028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2293.640743] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2300.392918] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2303.273993] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2308.612065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2308.666649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2314.132887] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2314.324087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2314.490339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2321.200950] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2324.930979] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2325.183071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2328.633225] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2328.832722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2335.201922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.205845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2335.226948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.235364] Lustre: Skipped 7 previous similar messages [ 2336.637099] LustreError: 90333:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2336.639032] LustreError: 90333:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2336.838375] Lustre: server umount lustre-OST0000 complete [ 2346.572292] Lustre: server umount lustre-MDT0000 complete [ 2350.065080] LustreError: 87388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491925 with bad export cookie 4269339896587562093 [ 2350.065617] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2350.080688] LustreError: 87388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2350.560961] Lustre: server umount lustre-MDT0001 complete [ 2359.431452] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 00:05:33 (1762491933) [ 2365.940836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.460160] LustreError: 91412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.006879] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2379.934725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2383.567567] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2389.111994] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2392.287421] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2398.391183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2403.307855] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2409.150677] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2413.866824] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2414.230220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2418.363923] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2418.594674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2425.577763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2425.627942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2425.723378] Lustre: lustre-OST0001: new disk, initializing [ 2425.725426] Lustre: Skipped 2 previous similar messages [ 2425.731353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2425.735632] Lustre: Skipped 3 previous similar messages [ 2425.803207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2425.807708] Lustre: Skipped 5 previous similar messages [ 2431.730172] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2434.685593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2434.696594] Lustre: Skipped 2 previous similar messages [ 2434.708699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2434.817471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2438.958632] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2443.086501] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2443.275163] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2446.710239] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2446.914097] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2463.744873] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2492.913316] Lustre: Failing over lustre-MDT0000 [ 2493.272306] LustreError: 96320:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2493.275762] LustreError: 96320:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2493.363900] Lustre: server umount lustre-MDT0000 complete [ 2495.297600] LustreError: 94697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.319351] LustreError: 94697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2496.483639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.487429] Lustre: Skipped 5 previous similar messages [ 2511.844779] Lustre: 64494:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762492071/real 1762492071] req@00000000bbae5784 x1848106099655808/t0(0) o400->MGC192.168.204.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1762492087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2511.889514] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.943073] LDISKFS-fs (dm-0): recovery complete [ 2517.947753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.458828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2525.968056] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2525.970308] Lustre: 91408:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2526.103383] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2527.713068] Lustre: 92124:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2527.728299] Lustre: 92124:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 2527.746852] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2527.840575] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2528.191138] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2528.522074] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2528.525463] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2528.839105] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2528.844387] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2528.865092] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2528.885430] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 7 previous similar messages [ 2529.585931] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2529.598072] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2529.927263] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2529.938808] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2530.928057] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2530.945768] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 5 previous similar messages [ 2531.586141] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2531.592725] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 5 previous similar messages [ 2532.239744] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2532.246992] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 2535.121594] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2535.139684] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 12 previous similar messages [ 2535.788093] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2535.802476] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2536.439529] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2536.451963] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2543.168825] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2543.182657] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 24 previous similar messages [ 2543.822147] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2543.825439] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 24 previous similar messages [ 2544.455605] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2544.468508] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 24 previous similar messages [ 2559.449192] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2559.478506] Lustre: 97068:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 50 previous similar messages [ 2560.132229] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2560.156022] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 50 previous similar messages [ 2560.495095] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2560.499243] LustreError: 97068:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 49 previous similar messages [ 2585.215131] Lustre: 97068:0:(ldlm_lib.c:2375:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2585.220486] Lustre: 97068:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2585.235817] Lustre: 97068:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0495042b-899c-4123-a235-bd1b152597cd@192.168.204.3@tcp [ 2585.244521] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2585.248599] Lustre: 97068:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@000000000b826e39 x1848106083749760/t0(8589934776) o35->0495042b-899c-4123-a235-bd1b152597cd@192.168.204.3@tcp:541/0 lens 392/0 e 0 to 0 dl 1762492171 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2585.269679] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2585.276073] Lustre: lustre-MDT0000: Denying connection for new client 0495042b-899c-4123-a235-bd1b152597cd (at 192.168.204.3@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 2585.288455] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2585.334749] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2585.337445] Lustre: Skipped 2 previous similar messages [ 2585.338828] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2585.371493] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2585.372023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 2594.281373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2594.285239] LustreError: Skipped 1 previous similar message [ 2594.287372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.292311] Lustre: Skipped 1 previous similar message [ 2594.297083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.304903] Lustre: Skipped 5 previous similar messages [ 2599.400837] LustreError: 93009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.413327] Lustre: server umount lustre-OST0000 complete [ 2599.427972] LustreError: 93009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 2608.915630] Lustre: server umount lustre-OST0001 complete [ 2620.440273] Lustre: server umount lustre-MDT0000 complete [ 2623.620958] LustreError: 91392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492198 with bad export cookie 4269339896587613312 [ 2623.627522] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2623.628620] LustreError: 91392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2623.731351] LustreError: 98334:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2623.734460] LustreError: 98334:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 2623.976739] Lustre: server umount lustre-MDT0001 complete [ 2632.880217] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 00:10:06 (1762492206) [ 2640.984178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.999513] Lustre: *** cfs_fail_loc=197, val=0*** [ 2656.801864] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2661.919466] LustreError: 99025:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.103@tcp: failed processing log, type 4: rc = -110 [ 2667.074705] Lustre: *** cfs_fail_loc=197, val=0*** [ 2667.084967] Lustre: Skipped 65 previous similar messages [ 2687.650430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.652618] Lustre: Skipped 1 previous similar message [ 2692.696550] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2697.837477] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2699.699791] Lustre: server umount lustre-OST0000 complete [ 2706.332837] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 00:11:20 (1762492280) [ 2726.910203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.456262] LustreError: 100546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.480046] LustreError: 100546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2731.445382] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2738.850157] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2742.573725] LustreError: 100576:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2742.584607] LustreError: 100576:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2742.595656] LustreError: 100576:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2742.972628] Lustre: server umount lustre-MDT0000 complete [ 2756.445957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.802135] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 2779.127857] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 2787.844288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2794.887900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2800.907814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2807.639492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.676148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.750318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.062221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2815.102803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2815.175635] Lustre: lustre-MDT0000: new disk, initializing [ 2815.336085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2819.567379] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2830.132595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2830.204498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2830.298975] Lustre: 104615:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2830.687802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2834.432678] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2841.654858] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2844.780210] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2850.520231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2850.561482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2855.079413] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2860.197735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2861.809094] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2865.388931] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2865.608959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2869.155640] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2869.336790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2870.755655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2870.756575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.760302] LustreError: Skipped 2 previous similar messages [ 2870.762546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2870.769534] Lustre: Skipped 6 previous similar messages [ 2870.771500] Lustre: Skipped 5 previous similar messages [ 2885.087188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2885.138215] LustreError: 107554:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2885.150917] LustreError: 107554:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2885.364700] Lustre: server umount lustre-OST0000 complete [ 2895.863113] Lustre: server umount lustre-MDT0000 complete [ 2900.289641] LustreError: 104608:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492475 with bad export cookie 4269339896587623945 [ 2900.295201] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2900.313407] LustreError: 104608:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2900.803761] Lustre: server umount lustre-MDT0001 complete [ 2909.438715] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 00:14:43 (1762492483) [ 2918.754478] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 2921.288686] Key type lgssc unregistered [ 2921.595822] LNet: 108689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.601935] LNetError: 108689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2921.617970] LNet: Removed LNI 192.168.204.103@tcp [ 2922.449089] Key type .llcrypt unregistered [ 2922.450412] Key type ._llcrypt unregistered [ 2936.350069] Key type ._llcrypt registered [ 2936.351781] Key type .llcrypt registered [ 2936.416415] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 2948.425072] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 2949.320240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2949.371714] alg: No test for adler32 (adler32-zlib) [ 2950.457596] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 2950.673832] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 2952.311215] Key type lgssc registered [ 2953.262174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2962.595924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2968.784310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2974.693337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2980.690992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2985.844991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.515680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.600568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2992.615088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.795972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2993.834290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2993.889896] Lustre: lustre-MDT0000: new disk, initializing [ 2993.972566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.989594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2996.885994] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3005.516705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3005.603467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3005.863705] Lustre: lustre-OST0000: new disk, initializing [ 3005.869088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3005.884037] Lustre: Skipped 1 previous similar message [ 3005.987659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.989095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3007.995973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3008.022306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3010.815569] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3019.687194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3019.761617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3019.912357] Lustre: lustre-OST0001: new disk, initializing [ 3019.924119] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3019.998863] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.767147] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3026.574491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3026.587487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3026.615521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3037.158838] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.165947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.176532] Lustre: Skipped 1 previous similar message [ 3038.432918] LustreError: 114248:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 3038.754722] Lustre: server umount lustre-MDT0000 complete [ 3044.511320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3048.954938] LustreError: 114162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492624 with bad export cookie 14386009976009990434 [ 3048.963321] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.978776] LustreError: 114162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3049.032407] LustreError: 114843:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3049.043990] LustreError: 114843:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3049.103810] Lustre: server umount lustre-OST0000 complete [ 3052.668562] LustreError: 115045:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3052.671674] LustreError: 115045:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3052.926575] Lustre: server umount lustre-OST0001 complete [ 3062.895399] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 00:17:16 (1762492636) [ 3072.584889] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3080.834122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3081.866365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3086.359436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3087.388842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3093.307969] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 00:17:47 (1762492667) [ 3104.161156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3105.834744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3107.837118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3109.952082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3110.838935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3112.589663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3127.376700] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 3138.479649] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3149.570331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3155.221080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3160.830549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3167.343817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.194050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.246085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.429220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3174.490516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3174.570713] Lustre: lustre-MDT0000: new disk, initializing [ 3174.705499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.729382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3178.427119] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3188.340968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3188.424256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3188.498540] Lustre: 120372:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3188.568612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3188.572557] Lustre: Skipped 1 previous similar message [ 3188.624994] Lustre: lustre-MDT0001: new disk, initializing [ 3188.731576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3188.782659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3188.791972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3192.186954] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3199.146381] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3202.119676] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3207.647727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3207.706806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3207.961085] Lustre: lustre-OST0000: new disk, initializing [ 3207.967839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3208.097710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.471643] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3216.998551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3217.009382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3217.131625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3220.316762] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3224.383469] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3224.713126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3228.352840] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3228.569309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3232.739580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3232.742198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.744091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.744098] Lustre: Skipped 1 previous similar message [ 3232.745484] LustreError: Skipped 1 previous similar message [ 3232.780639] Lustre: Skipped 1 previous similar message [ 3236.351141] LustreError: 123311:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3236.355618] LustreError: 123311:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3236.576795] Lustre: server umount lustre-OST0000 complete [ 3242.984224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3243.008184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3243.012047] Lustre: Skipped 2 previous similar messages [ 3246.328611] LustreError: 123515:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3246.356859] LustreError: 123515:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3246.473522] Lustre: server umount lustre-MDT0000 complete [ 3249.715857] LustreError: 120367:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492824 with bad export cookie 14386009976009991659 [ 3249.717566] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3249.735608] LustreError: 120367:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3250.175461] Lustre: server umount lustre-MDT0001 complete [ 3257.597634] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 00:20:31 (1762492831) [ 3263.572619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.961410] LustreError: 124392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.977898] LustreError: 124392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3264.027086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.163612] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3269.090374] LustreError: 124391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.100102] LustreError: 124391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3274.209362] LustreError: 124391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.340640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3277.630811] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3282.353089] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3285.841644] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3292.595255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3297.535485] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3302.027914] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3316.542106] Lustre: *** cfs_fail_loc=159, val=0*** [ 3332.915813] Lustre: lustre-MDT0000: Client f8135e77-ccce-4b8d-9909-6e591de3b318 (at 192.168.204.3@tcp) reconnecting [ 3334.324615] Lustre: *** cfs_fail_loc=159, val=0*** [ 3334.333919] Lustre: Skipped 5 previous similar messages [ 3349.853239] Lustre: lustre-MDT0000: Client f8135e77-ccce-4b8d-9909-6e591de3b318 (at 192.168.204.3@tcp) reconnecting [ 3355.619887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.623833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3355.636567] Lustre: Skipped 1 previous similar message [ 3355.651102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3355.653511] Lustre: Skipped 1 previous similar message [ 3359.173336] LustreError: 126906:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3359.176366] LustreError: 126906:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3359.398307] Lustre: server umount lustre-OST0000 complete [ 3365.859581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.863758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3365.879377] Lustre: Skipped 2 previous similar messages [ 3365.905901] Lustre: Skipped 1 previous similar message [ 3369.215410] Lustre: server umount lustre-MDT0000 complete [ 3370.988691] LustreError: 124387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.016718] LustreError: 124387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3373.771111] LustreError: 126804:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492949 with bad export cookie 14386009976009992541 [ 3373.774109] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3373.789140] LustreError: 126804:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3374.411482] Lustre: server umount lustre-MDT0001 complete [ 3384.830949] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3387.786795] Key type lgssc unregistered [ 3388.069667] LNet: 127840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3388.086757] LNetError: 127840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3388.108693] LNet: Removed LNI 192.168.204.103@tcp [ 3388.909364] Key type .llcrypt unregistered [ 3388.912755] Key type ._llcrypt unregistered [ 3410.557906] Key type ._llcrypt registered [ 3410.559248] Key type .llcrypt registered [ 3410.688575] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3411.609525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3411.623545] alg: No test for adler32 (adler32-zlib) [ 3412.677157] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3412.886629] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3414.551173] Key type lgssc registered [ 3415.482514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3421.859482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3421.884849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.328427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.133718] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3433.442446] LustreError: 129233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.447198] LustreError: 129233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3433.997472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3434.243489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3437.393742] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3441.690251] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3444.949487] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3451.520604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3452.053909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.555381] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3461.385589] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3468.477907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3479.170492] Lustre: *** cfs_fail_loc=159, val=0*** [ 3494.736475] Lustre: lustre-MDT0000: Client 426adb94-bb20-444f-87a7-5a2a06625d96 (at 192.168.204.3@tcp) reconnecting [ 3497.831642] Lustre: *** cfs_fail_loc=159, val=0*** [ 3514.166042] Lustre: lustre-MDT0001: Client 426adb94-bb20-444f-87a7-5a2a06625d96 (at 192.168.204.3@tcp) reconnecting [ 3515.566366] Lustre: *** cfs_fail_loc=159, val=0*** [ 3515.568125] Lustre: Skipped 3 previous similar messages [ 3532.094165] Lustre: lustre-MDT0001: Client 426adb94-bb20-444f-87a7-5a2a06625d96 (at 192.168.204.3@tcp) reconnecting [ 3535.202610] Lustre: 131961:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3537.549538] Lustre: *** cfs_fail_loc=159, val=0*** [ 3537.554214] Lustre: Skipped 4 previous similar messages [ 3554.079663] Lustre: lustre-MDT0000: Client 9a1ac3bb-f6b9-499a-b643-e0be127c1881 (at 192.168.204.3@tcp) reconnecting [ 3555.425150] Lustre: *** cfs_fail_loc=159, val=0*** [ 3555.427185] Lustre: Skipped 14 previous similar messages [ 3572.009903] Lustre: lustre-MDT0000: Client 9a1ac3bb-f6b9-499a-b643-e0be127c1881 (at 192.168.204.3@tcp) reconnecting [ 3573.325108] Lustre: 132246:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3578.856143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3578.864358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.869314] LustreError: Skipped 1 previous similar message [ 3578.915022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3581.408948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3581.409248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3581.410800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3581.410807] LustreError: Skipped 3 previous similar messages [ 3584.112221] LustreError: 132344:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3584.253888] Lustre: server umount lustre-OST0000 complete [ 3588.064394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3588.069179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.085059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.092297] Lustre: Skipped 1 previous similar message [ 3591.649831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.655461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.668052] Lustre: Skipped 1 previous similar message [ 3593.619960] LustreError: 132545:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3593.629175] LustreError: 132545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3593.734355] Lustre: server umount lustre-MDT0000 complete [ 3596.771326] LustreError: 129233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.797341] LustreError: 129233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3597.149512] LustreError: 129214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493172 with bad export cookie 17556417316303969242 [ 3597.155630] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3597.155904] LustreError: 129214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3597.266598] LustreError: 132748:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3597.269017] LustreError: 132748:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3597.594869] Lustre: server umount lustre-MDT0001 complete [ 3608.606244] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3611.473475] Key type lgssc unregistered [ 3611.836838] LNet: 133280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3611.852800] LNetError: 133280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3611.874363] LNet: Removed LNI 192.168.204.103@tcp [ 3612.765127] Key type .llcrypt unregistered [ 3612.775945] Key type ._llcrypt unregistered [ 3636.407419] Key type ._llcrypt registered [ 3636.425710] Key type .llcrypt registered [ 3636.610711] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3637.823564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3637.834546] alg: No test for adler32 (adler32-zlib) [ 3638.953793] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3639.302130] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3640.983160] Key type lgssc registered [ 3642.457303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3650.744800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3650.762774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.269120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.095786] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3661.793541] LustreError: 134677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.809904] LustreError: 134677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3664.139054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3664.401285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3667.645354] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3672.014203] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3675.046908] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3681.104917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3681.389094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.981483] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3689.617763] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3692.494513] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3697.819536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3703.722750] Lustre: 136933:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3706.334646] Lustre: 136982:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3707.554942] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3708.238538] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3710.678533] Lustre: 137127:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3711.285808] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3711.976730] Lustre: 137226:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3715.552946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3715.563095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.583220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.625548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.626508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.635728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3720.826639] LustreError: 137323:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3720.957807] Lustre: server umount lustre-OST0000 complete [ 3725.794387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3725.798113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.823553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.835451] Lustre: Skipped 1 previous similar message [ 3728.865976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3728.868182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.072782] LustreError: 137531:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3730.100961] LustreError: 137531:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3730.190369] Lustre: server umount lustre-MDT0000 complete [ 3733.824096] LustreError: 136296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493309 with bad export cookie 8684671810804707794 [ 3733.825858] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3733.844155] LustreError: 136296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3733.975537] LustreError: 137733:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3733.984489] LustreError: 137733:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3734.160092] Lustre: server umount lustre-MDT0001 complete [ 3743.514644] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3745.830233] Key type lgssc unregistered [ 3746.112658] LNet: 138264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3746.126590] LNetError: 138264:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3746.135822] LNet: Removed LNI 192.168.204.103@tcp [ 3746.962444] Key type .llcrypt unregistered [ 3746.964541] Key type ._llcrypt unregistered [ 3764.482056] Key type ._llcrypt registered [ 3764.488238] Key type .llcrypt registered [ 3764.574783] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3765.336662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3765.377854] alg: No test for adler32 (adler32-zlib) [ 3766.428593] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3766.727671] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3768.408879] Key type lgssc registered [ 3769.424600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3776.300909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3776.321921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3777.656261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.797728] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3787.238684] LustreError: 139639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.271440] LustreError: 139639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3789.348807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3790.244959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3794.612764] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3801.538506] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3805.766845] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3813.175661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3813.799045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.835551] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 3820.500239] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3822.000096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 3825.917719] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3832.058046] Lustre: *** cfs_fail_loc=159, val=0*** [ 3832.068176] Lustre: Skipped 1 previous similar message [ 3847.468354] Lustre: lustre-MDT0000: Client 190be28c-7aaa-4617-9ae9-0db681cfb603 (at 192.168.204.3@tcp) reconnecting [ 3852.767802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.781201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3852.783628] Lustre: Skipped 1 previous similar message [ 3853.500186] LustreError: 142017:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3853.645595] Lustre: server umount lustre-OST0000 complete [ 3858.380928] LustreError: 142218:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3858.386769] LustreError: 142218:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3858.612958] Lustre: server umount lustre-MDT0000 complete [ 3861.994336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3861.997501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3862.029353] LustreError: 139638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.020454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.041037] LustreError: 141852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.077706] LustreError: 141852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3863.223558] LustreError: 139619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493438 with bad export cookie 4701919348322647160 [ 3863.250764] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3865.455431] LustreError: 142419:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3865.462304] LustreError: 142419:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3865.736118] Lustre: server umount lustre-MDT0001 complete [ 3875.678922] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3878.077485] Key type lgssc unregistered [ 3878.329714] LNet: 142951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3878.334823] LNetError: 142951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3878.367877] LNet: Removed LNI 192.168.204.103@tcp [ 3879.211241] Key type .llcrypt unregistered [ 3879.215959] Key type ._llcrypt unregistered [ 3898.791479] Key type ._llcrypt registered [ 3898.794156] Key type .llcrypt registered [ 3898.884824] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3899.784582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3899.877476] alg: No test for adler32 (adler32-zlib) [ 3900.984136] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3901.223799] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3902.887181] Key type lgssc registered [ 3904.074857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3911.440657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3911.473152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.934927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3916.929116] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3923.427822] LustreError: 144327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3923.435334] LustreError: 144327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3924.906038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3925.555627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3930.044799] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3936.159629] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3940.151539] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3948.098634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3948.484490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3950.513077] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 3954.922744] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3958.727971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 3959.959318] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3979.899050] Lustre: lustre-OST0000: evicting bec2a315-0109-4dc3-a819-ef7714f9a9ce (at 192.168.204.3@tcp) by administrative request [ 3979.913940] Lustre: Skipped 1 previous similar message [ 4009.952236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.956631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.967981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.995496] Lustre: Skipped 1 previous similar message [ 4014.682328] LustreError: 147125:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4014.829633] Lustre: server umount lustre-OST0000 complete [ 4020.200651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.207648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.209389] Lustre: Skipped 1 previous similar message [ 4020.217872] Lustre: Skipped 2 previous similar messages [ 4024.346305] LustreError: 147326:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4024.348906] LustreError: 147326:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4024.475613] Lustre: server umount lustre-MDT0000 complete [ 4025.313562] LustreError: 144327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.705131] LustreError: 144306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493602 with bad export cookie 17218599554631010180 [ 4027.705385] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4027.717730] LustreError: 144306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4027.863862] LustreError: 147528:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4027.869453] LustreError: 147528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4028.056886] Lustre: server umount lustre-MDT0001 complete [ 4039.147416] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 4042.610566] Key type lgssc unregistered [ 4043.110929] LNet: 148061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4043.125147] LNetError: 148061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4044.138034] LNet: Removed LNI 192.168.204.103@tcp [ 4045.246338] Key type .llcrypt unregistered [ 4045.249062] Key type ._llcrypt unregistered [ 4111.394874] Key type ._llcrypt registered [ 4111.399517] Key type .llcrypt registered [ 4111.563600] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 4112.351220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4112.377549] alg: No test for adler32 (adler32-zlib) [ 4113.580814] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 4113.880504] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 4115.591241] Key type lgssc registered [ 4116.367878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4121.433507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4121.453787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.784735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.609391] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4131.958660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4132.269203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4135.012372] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4138.590275] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4141.287947] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.586232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4146.911277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.954523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4151.028787] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4154.711307] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4157.170807] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4162.530415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4162.546393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.550914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4162.600209] Lustre: Skipped 1 previous similar message [ 4164.364340] LustreError: 152783:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4164.632811] Lustre: server umount lustre-OST0000 complete [ 4172.769819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.773075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.781261] Lustre: Skipped 1 previous similar message [ 4172.796020] Lustre: Skipped 1 previous similar message [ 4174.922907] LustreError: 152985:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4174.972574] LustreError: 152985:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4175.131865] Lustre: server umount lustre-MDT0000 complete [ 4177.905533] LustreError: 150525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.918468] LustreError: 150525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4179.326806] LustreError: 150508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493754 with bad export cookie 9557585345296941748 [ 4179.334712] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4179.345811] LustreError: 150508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4179.502961] LustreError: 153187:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4179.510714] LustreError: 153187:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4179.864655] Lustre: server umount lustre-MDT0001 complete [ 4187.468190] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 4189.600246] Key type lgssc unregistered [ 4189.824682] LNet: 153718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4189.833849] LNetError: 153718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4190.887522] LNet: Removed LNI 192.168.204.103@tcp [ 4191.634111] Key type .llcrypt unregistered [ 4191.635693] Key type ._llcrypt unregistered [ 4216.853571] Key type ._llcrypt registered [ 4216.855111] Key type .llcrypt registered [ 4216.962571] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 4218.200583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4218.299079] alg: No test for adler32 (adler32-zlib) [ 4219.308521] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 4219.523447] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 4221.175183] Key type lgssc registered [ 4222.436819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4227.822228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4234.134094] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 00:36:48 (1762493808) [ 4248.314782] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 4257.378197] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 4267.444921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4274.977634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4281.376086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4288.527993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4295.986074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4296.051837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4296.072948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.424545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4297.488763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4297.613857] Lustre: lustre-MDT0000: new disk, initializing [ 4297.722942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.751485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4301.559711] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4311.987719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4312.049080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4312.113984] Lustre: 157944:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4312.186240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4312.189108] Lustre: Skipped 1 previous similar message [ 4312.273474] Lustre: lustre-MDT0001: new disk, initializing [ 4312.396249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4312.446287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4312.462535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4316.264914] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4324.641371] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4329.325650] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4337.491665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4337.574831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4337.811835] Lustre: lustre-OST0000: new disk, initializing [ 4337.816438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4337.906619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.657516] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4347.942860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4347.948921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4348.048940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4350.470638] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4363.240111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.246119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4363.253564] Lustre: Skipped 1 previous similar message [ 4363.274447] Lustre: Skipped 1 previous similar message [ 4364.822122] LustreError: 160532:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4365.038871] Lustre: server umount lustre-OST0000 complete [ 4368.867247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4368.880761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.896740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.472032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.474438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.814883] LustreError: 160733:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4374.818939] LustreError: 160733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4374.937879] Lustre: server umount lustre-MDT0000 complete [ 4378.395338] LustreError: 157934:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493953 with bad export cookie 7900972012139886465 [ 4378.401877] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.510417] LustreError: 160935:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4378.524765] LustreError: 160935:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4378.816743] Lustre: server umount lustre-MDT0001 complete [ 4389.763850] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 4392.807114] Key type lgssc unregistered [ 4393.173832] LNet: 161469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4393.196138] LNetError: 161469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4393.211651] LNet: Removed LNI 192.168.204.103@tcp [ 4394.284149] Key type .llcrypt unregistered [ 4394.288790] Key type ._llcrypt unregistered [ 4420.219955] Key type ._llcrypt registered [ 4420.221286] Key type .llcrypt registered [ 4420.392844] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 4421.291563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4421.312737] alg: No test for adler32 (adler32-zlib) [ 4422.404407] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 4422.684958] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 4424.335174] Key type lgssc registered [ 4425.485752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4431.838078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4431.864924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.349058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.713612] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4443.634939] LustreError: 162867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.681972] LustreError: 162867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4445.016986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4445.455568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4449.357559] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4454.484468] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4457.927138] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4464.035625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4464.285898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.891659] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4475.122580] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4478.645724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4485.471773] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4485.499334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.512130] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4485.527486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4485.553071] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4486.507884] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4486.522442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.530271] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4486.533913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4486.548714] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4487.664780] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4487.685365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.696636] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4487.700339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4487.711805] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4488.768962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4489.562738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4490.574706] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4490.578381] Lustre: Skipped 2 previous similar messages [ 4490.618322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.624686] Lustre: Skipped 2 previous similar messages [ 4490.626577] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4490.629697] Lustre: Skipped 2 previous similar messages [ 4490.631633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4490.639971] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4490.642674] Lustre: Skipped 2 previous similar messages [ 4491.543639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4492.393310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4493.197796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.052715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.998457] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4495.010044] Lustre: Skipped 4 previous similar messages [ 4495.021611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4495.027026] Lustre: Skipped 4 previous similar messages [ 4495.030810] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4495.034195] Lustre: Skipped 4 previous similar messages [ 4495.038656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4495.053199] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4495.057136] Lustre: Skipped 4 previous similar messages [ 4495.793797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4496.560112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4497.337524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.011142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.763450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4499.502728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4500.268058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4501.073102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4501.868444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4502.633037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4503.474531] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4503.477387] Lustre: Skipped 10 previous similar messages [ 4503.502832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.509888] Lustre: Skipped 10 previous similar messages [ 4503.514168] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4503.518675] Lustre: Skipped 10 previous similar messages [ 4503.521571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4503.547224] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4503.550353] Lustre: Skipped 10 previous similar messages [ 4504.241430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.020246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.781388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4506.560512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4507.275467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4508.080323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4508.653467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.160137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.724211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4510.345634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4510.980459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.607440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.245502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.887143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4513.592736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4514.279761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.116372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.795392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4516.513508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4517.209897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.015781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.869931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.640250] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4519.646532] Lustre: Skipped 22 previous similar messages [ 4519.665740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.671413] Lustre: Skipped 22 previous similar messages [ 4519.676081] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4519.679508] Lustre: Skipped 22 previous similar messages [ 4519.682768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.692415] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4519.695432] Lustre: Skipped 22 previous similar messages [ 4520.405375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4521.100806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4521.810522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4522.488092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.203767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.853380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4524.563309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4525.284920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4526.097151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4526.789290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4527.543073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4528.266350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4529.316149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4530.148626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4530.991668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4531.896367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4532.702177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4533.460208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4534.344800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4535.214781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4536.170655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4537.029204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4537.870746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4538.626777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.371868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.995661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4540.526300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4541.136778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4541.737707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4542.273770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4542.967098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4543.593780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.300881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.946388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4545.529469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4546.130326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4546.864710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.519405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4548.229575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4549.038945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4549.793783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4550.445153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4551.211930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4551.972952] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4551.976227] Lustre: Skipped 43 previous similar messages [ 4551.990495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.999315] Lustre: Skipped 43 previous similar messages [ 4552.005449] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4552.008360] Lustre: Skipped 43 previous similar messages [ 4552.013619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4552.024771] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4552.028494] Lustre: Skipped 43 previous similar messages [ 4552.738086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4553.451244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4554.231926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4554.798562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4555.395374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4556.121486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4556.791836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4557.449980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.027798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.563622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4559.160480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4559.781940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4560.474767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4561.210742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4561.893714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4562.753504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4563.559521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4564.305545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4565.008760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4565.780880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4566.621236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4567.371947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4568.029327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4568.863277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4569.655496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4570.500200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4571.163214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4571.926483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4572.696851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4573.472473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4574.204750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4574.988539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4576.020732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4576.795822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4577.522564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4578.301497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4579.080349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4579.831037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4580.579306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4581.535532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.279504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4583.433186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4584.324210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4585.241309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4586.129467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4587.061477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4587.957301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4588.899083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4589.836553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4590.779874] LustreError: 162886:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4590.785580] LustreError: 162886:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4590.792217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4591.838261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4592.680039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4593.712184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4594.664164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4595.550117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4596.470414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4597.278639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4598.155463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4598.899418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4600.062506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4600.970151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4601.976232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4602.892296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4603.807464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4604.635859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4605.537732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4606.384619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4607.154579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4607.894709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4608.683203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4609.636592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4610.380310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4611.089923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4611.911924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4612.737192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4613.653326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4614.423056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4615.176127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4616.000203] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4616.004704] Lustre: Skipped 78 previous similar messages [ 4616.017796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4616.025639] Lustre: Skipped 78 previous similar messages [ 4616.029442] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4616.036253] Lustre: Skipped 78 previous similar messages [ 4616.043316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4616.055635] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4616.064042] Lustre: Skipped 78 previous similar messages [ 4616.971557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.807099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4618.668412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4619.505295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4620.342563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4621.357595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.539587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4623.417308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4624.339337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4625.227866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4626.033285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4626.815867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4627.528529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4628.283398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4629.109359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4629.956722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4630.815122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4631.596326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4632.459264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4633.140394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4634.084968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4634.992843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4635.941729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4636.718988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4637.433035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4638.259222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4639.202191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4640.069582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4640.953341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4641.964402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4642.913890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4643.930532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4644.783343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4645.639234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4646.300811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4646.964325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4647.728883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4648.347892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4649.234161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4650.028955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4651.238517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4652.236370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4653.153616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4654.221703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4655.113712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4656.036823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4657.590925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4659.123087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4660.539164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4661.913573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4662.674275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4663.803511] LustreError: 162886:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4663.820168] LustreError: 162886:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4663.836915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4665.049988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4666.081411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4667.465581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4668.865558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4670.292075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4671.987080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4673.637461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4674.945435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4676.151743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4677.063736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4678.014985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4678.968994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4679.842284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4680.817458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4681.810539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4682.725702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4683.907716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4684.852071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4685.612881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4686.413623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4687.204078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4687.898425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4688.745921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4689.512977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4690.481300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4691.411886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4692.308202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4693.107989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4694.080664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4695.016395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4695.875114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4696.780086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4697.618036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4698.468291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4699.380432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4700.280676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4701.164427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4701.912560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4702.788260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4703.558527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4704.580682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4705.491049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4706.408516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4707.284557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4708.152049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4708.902534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4709.752844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4710.708282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4711.626164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4712.553390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4713.552146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4714.561790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4715.352796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4716.201995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4717.207348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4718.143579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4718.973910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4719.845379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4720.800341] LustreError: 162886:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4720.809228] LustreError: 162886:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4720.815367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4721.651535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4722.558858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4723.424438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4724.123814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4724.961461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4725.900448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4726.714271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4727.658262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.498870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4729.418346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4730.307609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4731.248172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4732.200074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4733.183204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4734.194137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4735.229172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4735.986056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4736.750780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4737.599328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4738.433374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4739.232478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4740.137955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4740.976955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4741.912693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4742.727266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4743.604729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.452177] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4744.456307] Lustre: Skipped 137 previous similar messages [ 4744.468530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4744.475323] Lustre: Skipped 137 previous similar messages [ 4744.480828] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4744.483755] Lustre: Skipped 137 previous similar messages [ 4744.490217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.504198] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4744.510477] Lustre: Skipped 137 previous similar messages [ 4745.289726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4746.094160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4746.905115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4747.679799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4748.618144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4749.543961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4750.570795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4751.378332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4752.270287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4753.185785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4754.161295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4755.031659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4755.918463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4756.855121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.864491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4758.786587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.710271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4760.526186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4761.513515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4762.426565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4763.490272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.347558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4765.183184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.034742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4767.242816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.164893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.129091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.082717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.843050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.656443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4772.543495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.275263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.040536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.927095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.904471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.738317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.686261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.678711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4779.716907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.500267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.319844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.165964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.067906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.280683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.505668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4786.505512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4787.555370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.484861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.387199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.248247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4791.069620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.052889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.013457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.020370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.911305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.832866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.615498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.483081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.347699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.372082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.246724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.199029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.039630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.943988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.834221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.597852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4805.374457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.040265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.741702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.653397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.662459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.590184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4810.550390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.455465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.448077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.349482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.378394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.661345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.573742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.556381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.565110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.473290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4820.413660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.128416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.952438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.745545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.721120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4824.788283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.491912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.364836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.174108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.124648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.936822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4829.748527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.473081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4831.573846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.567723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.638918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.078648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4836.075988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.099773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.142402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4839.412354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.447745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.514328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.338970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.621694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4844.642205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.948498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4847.188709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.115294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.934258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.837927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.749498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.579272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4852.556315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.635776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.449756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.201526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4856.044841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4857.180418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.125612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4859.265708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.347079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.324137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4862.207219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.124328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4864.064734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4865.110549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.092511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.077837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4868.084938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.084638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4870.094787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.137243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4872.208147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.204210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.987680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.919808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4875.816106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.656444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4877.606499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4878.720454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.762335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.836098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4881.809565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.569926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4883.494632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.407730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.263091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4886.002872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4886.834962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.797938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.633131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.627322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.608551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4891.584418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.517607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.351601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4894.235830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.212604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.160411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.936462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4897.763745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.468107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4899.352768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4900.303714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.167614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4902.071573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4903.160467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4904.101971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4905.158572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4906.223982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.259935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4908.335225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4909.176989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.154839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4911.154040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4912.148631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4912.984254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.108569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4915.059241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.048648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.973172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.919135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.844577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4920.014948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.005304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4922.115474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.303435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4924.291496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.688468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.880382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.712371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4928.889341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.918107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.961978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.909631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.956345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.966439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.110353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4936.102421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.141765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.267398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.411634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4940.569504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.565586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.689567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.587329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.742232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.982706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4946.903991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.777957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.685246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.592954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.446264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.541119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.582250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.679673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.785990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.949383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.862161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.758800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4958.725332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.663666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.755711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.754859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.135777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.173220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.246689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.610677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.488360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.945485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.138087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.478790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.563445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.493385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.728288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.740713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.848032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.949186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.779375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.609857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.686682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.703964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.712918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.741574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.922484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.887109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.845545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.858067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.856254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.772157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.592706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.541985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.491570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.386379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.340528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.230873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.271853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.205691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.134547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.018906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.093116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.196749] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5001.200425] Lustre: Skipped 264 previous similar messages [ 5001.212694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.228335] Lustre: Skipped 264 previous similar messages [ 5001.233602] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5001.235804] Lustre: Skipped 264 previous similar messages [ 5001.238991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.250187] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5001.252773] Lustre: Skipped 264 previous similar messages [ 5002.103699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.085258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.969334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.838023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.532080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.377671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.115343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.974324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.715305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.599186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.432965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.334853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.030770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.771446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.558350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.433710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.248708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.139656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.876441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.701979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.426592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.154423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.892908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.668268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.484429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.344397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.074238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.735452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.498143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.311863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.090746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.866887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.657769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.459771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.243324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.063293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.850474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.682638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.576769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.346195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.091782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.848724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.601671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.205624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.830192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.519601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.132140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.832871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.698651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.514652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.333707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.225699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.130211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.139592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.238233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.293696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.162716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.099789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.891890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.785060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.778371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.741607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.660905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.704939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.678469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.782333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.822820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.717371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.909327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.927861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.931703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.937266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.093184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.991333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.950711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.772939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.675253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.903739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.785513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.600457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.426385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.371538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.353407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.217936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.099240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.236374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.265572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.247054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.821449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.649331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.323067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.254599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.029978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.762474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.416161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.136962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.839214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.581462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.327912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.040057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.790050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.721480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.508814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.334610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.916756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.640461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.235903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.921972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.602747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.613751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.369132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.207850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.003407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.776567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.691303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.528814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.469112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.304062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.176396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.263534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.339719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.319398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.496753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.592173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.495869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.475341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.308241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.219791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.931493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.867754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.899151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.884693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.759478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.908732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.710642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.826237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.648385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.696390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.801077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.682487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.666990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.579462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.598921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.535644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.543707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.351514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.328109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.259416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.373875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.473781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.444829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.511958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.283491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.218263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.116480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.117742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.859826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.734975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.468111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.285816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.154286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.027414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.003971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.728033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.582707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.361257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.270773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.915544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.452170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.131922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.727631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.297235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.986430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.608556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.242530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.910946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.740623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.438113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.219370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.094429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.846604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.623653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.452231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.255966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.043606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.807750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.615981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.337467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.233724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.095686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.806398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.410244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.263637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.960838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.921599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.774457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.536350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.384297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.299994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.090634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.872799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.658264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.510537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.313159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.176229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.009238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.796854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.489081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.191430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.931994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.683380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.412509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.149782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.915594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.705989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.388258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.157363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.866916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.828298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.646277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.571074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.469228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.390490] LustreError: 162886:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5189.403586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.403901] LustreError: 162886:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5190.301034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.204554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.198450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.090865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.995735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.791692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.722351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5322.741328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5325.793202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5325.797724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5328.313053] LustreError: 206427:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5328.505567] Lustre: server umount lustre-OST0000 complete [ 5336.041185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5336.049204] Lustre: Skipped 2 previous similar messages [ 5338.626881] LustreError: 206629:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5338.634025] LustreError: 206629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5338.756404] Lustre: server umount lustre-MDT0000 complete [ 5341.152333] LustreError: 162866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.163789] LustreError: 162866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5342.133748] LustreError: 163589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762494917 with bad export cookie 3361309482047774473 [ 5342.137565] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5342.160616] LustreError: 163589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5342.285792] LustreError: 206830:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5342.311569] LustreError: 206830:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5342.526416] Lustre: server umount lustre-MDT0001 complete [ 5350.888120] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 5352.823991] Key type lgssc unregistered [ 5353.123227] LNet: 207362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5353.138796] LNetError: 207362:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5353.156977] LNet: Removed LNI 192.168.204.103@tcp [ 5353.812967] Key type .llcrypt unregistered [ 5353.815766] Key type ._llcrypt unregistered [ 5372.997897] Key type ._llcrypt registered [ 5373.000976] Key type .llcrypt registered [ 5373.100133] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5373.720904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5373.772105] alg: No test for adler32 (adler32-zlib) [ 5374.815844] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 5375.030554] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 5376.663187] Key type lgssc registered [ 5377.660854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5383.669580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5383.683929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.118972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5388.155398] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5395.311934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5395.428598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5395.547720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5398.495217] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5403.294351] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5407.062648] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5413.840569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5414.273359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.279737] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5424.057528] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5430.728970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13939 to 0x280000401:13985) [ 5437.766698] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5439.300728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.314642] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5439.321488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.333550] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5439.335022] LustreError: 208773:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 5440.992608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.997981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5440.999460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5441.019922] Lustre: Skipped 1 previous similar message [ 5441.033897] Lustre: Skipped 1 previous similar message [ 5444.906093] Lustre: lustre-OST0000: Not available for connect from 192.168.204.3@tcp (stopping) [ 5449.996095] Lustre: lustre-OST0000: Not available for connect from 192.168.204.3@tcp (stopping) [ 5449.998889] Lustre: Skipped 2 previous similar messages [ 5455.118512] Lustre: lustre-OST0000: Not available for connect from 192.168.204.3@tcp (stopping) [ 5455.126938] Lustre: Skipped 2 previous similar messages [ 5455.327192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5455.363244] LustreError: 211172:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5455.553365] Lustre: server umount lustre-OST0000 complete [ 5459.359265] LustreError: 208773:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 5489.294899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5489.832391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.564744] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5501.053508] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5506.216611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.251302] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.260870] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5510.844569] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec [ 5514.010445] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa01e48839800.ost_server_uuid 50 [ 5515.202138] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa01e48839800.ost_server_uuid in FULL state after 0 sec [ 5521.891343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5521.900857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5521.902209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5521.931768] Lustre: Skipped 1 previous similar message [ 5523.845115] LustreError: 212498:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5523.857353] LustreError: 212498:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5524.068934] Lustre: server umount lustre-OST0000 complete [ 5528.546877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5528.551363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.561531] Lustre: Skipped 1 previous similar message [ 5541.343160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5541.436058] LustreError: 212700:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5541.439149] LustreError: 212700:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5541.520628] Lustre: server umount lustre-MDT0000 complete [ 5542.369613] LustreError: 209961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.380271] LustreError: 209961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5544.626877] LustreError: 208734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495119 with bad export cookie 10457276310750650363 [ 5544.635445] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5544.643064] LustreError: 208734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5544.786941] LustreError: 212902:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5544.792169] LustreError: 212902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5544.954750] Lustre: server umount lustre-MDT0001 complete [ 5552.137851] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 5554.381588] Key type lgssc unregistered [ 5554.644687] LNet: 213433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5554.662487] LNetError: 213433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5554.742296] LNet: Removed LNI 192.168.204.103@tcp [ 5555.460058] Key type .llcrypt unregistered [ 5555.464625] Key type ._llcrypt unregistered [ 5581.089033] Key type ._llcrypt registered [ 5581.090517] Key type .llcrypt registered [ 5581.181776] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5582.004988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5582.097984] alg: No test for adler32 (adler32-zlib) [ 5583.107620] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 5583.310483] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 5584.959187] Key type lgssc registered [ 5586.032218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5593.382102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5593.402593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.879374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.022975] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5605.347384] LustreError: 215129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.367258] LustreError: 215129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5605.400498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5605.760787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5608.947419] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5610.940380] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5615.225948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5615.462610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5620.157806] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5620.602721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13987 to 0x280000401:14017) [ 5627.408325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5627.470081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5627.582706] Lustre: lustre-OST0001: new disk, initializing [ 5627.585898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5627.678623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5632.113140] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5633.072212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5633.082068] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5633.175740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5640.298668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5646.088296] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5662.197454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5662.204177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.222231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5663.722332] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.724430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5663.732261] Lustre: Skipped 1 previous similar message [ 5663.739858] Lustre: Skipped 3 previous similar messages [ 5663.895818] LustreError: 218485:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5664.041994] Lustre: server umount lustre-MDT0000 complete [ 5667.772839] LustreError: 215108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495243 with bad export cookie 2104606058336980108 [ 5667.776479] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5667.796632] LustreError: 215108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5668.003788] LustreError: 218685:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5668.006433] LustreError: 218685:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5668.313976] Lustre: server umount lustre-MDT0001 complete [ 5682.777167] LustreError: 218886:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5682.779237] LustreError: 218886:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5682.837765] Lustre: server umount lustre-OST0000 complete [ 5685.727172] Lustre: 214391:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762495244/real 1762495244] req@ffff95191093d180 x1848110122881792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762495260 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5685.774370] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5685.808712] Lustre: Skipped 1 previous similar message [ 5687.005176] LustreError: 219100:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5687.007952] LustreError: 219100:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5687.168100] Lustre: server umount lustre-OST0001 complete [ 5689.833170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5689.940688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5690.853089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5690.929047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5691.938342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5692.019861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5692.855390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5692.916533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5704.450949] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5711.695459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5711.975581] LustreError: 220511:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.045827] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5715.166982] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5717.473600] LustreError: 220511:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.502445] LustreError: 220511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5721.854581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5722.162317] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5725.442352] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5727.683161] Lustre: 221620:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5733.047486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5738.045410] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5738.538953] LustreError: 221976:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5738.564114] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14049) [ 5743.595700] LustreError: 222354:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.879525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5745.115107] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5745.125830] Lustre: Skipped 1 previous similar message [ 5746.753223] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5750.061418] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5757.756657] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5761.565842] Lustre: 223468:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5778.405057] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5778.412838] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.426450] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5781.561486] LustreError: 223775:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5781.577631] LustreError: 223775:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5781.734661] Lustre: server umount mylustre-MDT0000 complete [ 5784.037965] LustreError: 220510:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5784.054482] LustreError: 220510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5784.734412] LustreError: 220486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495359 with bad export cookie 2104606058336982908 [ 5784.738318] LustreError: 220486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5784.742565] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5785.058316] Lustre: server umount mylustre-MDT0001 complete [ 5798.457315] LustreError: 224177:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5798.460515] LustreError: 224177:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5798.531074] Lustre: server umount mylustre-OST0000 complete [ 5813.005483] Lustre: server umount mylustre-OST0001 complete [ 5815.692231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5815.770119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5816.612840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5816.671390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5817.560279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5817.671112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5818.604506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5818.693044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.299577] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5837.685434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5838.143555] LustreError: 225791:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.260874] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.066990] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5850.303846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5854.748395] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5857.982375] Lustre: 226901:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5863.934428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5868.700373] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5875.567129] LustreError: 227258:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5875.573407] LustreError: 227258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5875.877173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5881.943717] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5883.636336] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14081) [ 5883.668500] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5888.242938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5891.168838] Lustre: 228749:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5904.363707] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.369561] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5904.377128] Lustre: Skipped 6 previous similar messages [ 5904.382980] Lustre: Skipped 6 previous similar messages [ 5908.476664] LustreError: 228997:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5908.492588] LustreError: 228997:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5908.581747] Lustre: server umount tfs-MDT0000 complete [ 5911.888929] LustreError: 225768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495487 with bad export cookie 2104606058336984805 [ 5911.904552] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5912.238504] Lustre: server umount tfs-MDT0001 complete [ 5925.767380] Lustre: server umount tfs-OST0000 complete [ 5940.110989] Lustre: server umount tfs-OST0001 complete [ 5942.866708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5942.942763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5943.812734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5943.900854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5944.901582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5945.011529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5945.855482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5945.927600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5957.424963] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5966.162273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.646234] LustreError: 231014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.662880] LustreError: 231014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5966.750756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.753865] Lustre: Skipped 3 previous similar messages [ 5970.285372] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5977.088025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5980.334247] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5982.575131] Lustre: 232122:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5988.514778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5990.965619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14113) [ 5993.747857] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6002.183405] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6003.982178] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6007.009370] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6014.269458] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6017.863206] Lustre: 233968:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6030.928487] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 01:06:44 (1762495604) [ 6048.745608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.755701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6048.764130] Lustre: Skipped 2 previous similar messages [ 6052.090110] LustreError: 234339:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6052.094127] LustreError: 234339:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6052.155985] Lustre: server umount lustre-MDT0000 complete [ 6055.897961] LustreError: 230988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495631 with bad export cookie 2104606058336986464 [ 6055.899308] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6055.913926] LustreError: 230988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6058.984159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6070.240116] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6070.537225] Lustre: server umount lustre-MDT0001 complete [ 6080.709914] Lustre: server umount lustre-OST0000 complete [ 6091.467436] Lustre: server umount lustre-OST0001 complete [ 6096.630288] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 6103.555133] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6114.489047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6122.983846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6129.935156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6135.748744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6141.788950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6141.846459] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 6141.846459] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6141.846459] [ 6141.863379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6142.128413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6142.164475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6142.224804] Lustre: lustre-MDT0000: new disk, initializing [ 6142.326280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6142.331444] Lustre: Skipped 3 previous similar messages [ 6142.344926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6146.378103] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6156.156871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6156.237203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6156.367292] Lustre: 237138:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6156.438505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6156.441560] Lustre: Skipped 1 previous similar message [ 6156.510648] Lustre: lustre-MDT0001: new disk, initializing [ 6156.667592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6156.681182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6159.995259] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6169.515082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6169.570848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6169.788972] Lustre: lustre-OST0000: new disk, initializing [ 6169.796099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6171.643957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6171.655256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6171.923839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6175.193240] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6184.622098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6184.705244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6184.885200] Lustre: lustre-OST0001: new disk, initializing [ 6184.897036] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6186.559466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6186.566222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6186.664255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6189.704300] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6199.619267] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 01:09:33 (1762495773) [ 6210.048346] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 01:09:44 (1762495784) [ 6227.958251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.960463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6227.962419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6227.962430] Lustre: Skipped 5 previous similar messages [ 6227.974479] Lustre: Skipped 6 previous similar messages [ 6230.508704] LustreError: 240560:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6230.529720] LustreError: 240560:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6230.633504] Lustre: server umount lustre-OST0000 complete [ 6233.063591] LustreError: 238674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.097929] LustreError: 238674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 6240.724073] Lustre: server umount lustre-MDT0000 complete [ 6244.783934] LustreError: 238688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495820 with bad export cookie 2104606058336988018 [ 6244.787661] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.799669] LustreError: 238688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6259.167297] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6259.433072] Lustre: server umount lustre-MDT0001 complete [ 6268.176901] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 6284.274416] Lustre: server umount lustre-OST0001 complete [ 6290.755755] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 6305.252527] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6314.648238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6321.346748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6327.421087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6334.180112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.469546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.543692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.792360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6341.841776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6342.001839] Lustre: lustre-MDT0000: new disk, initializing [ 6342.110619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6346.192941] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6356.442991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6356.513708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6356.598655] Lustre: 244050:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6356.840370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6360.828937] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6368.040422] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6371.257981] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6376.805755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.869722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6377.080958] Lustre: lustre-OST0000: new disk, initializing [ 6377.086368] Lustre: Skipped 1 previous similar message [ 6377.089822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6377.093742] Lustre: Skipped 2 previous similar messages [ 6378.246501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6378.256465] Lustre: Skipped 1 previous similar message [ 6378.261787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6378.367925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6382.298335] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6388.498637] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6393.826279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6393.831679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.840430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6393.858169] Lustre: Skipped 5 previous similar messages [ 6393.861635] Lustre: Skipped 8 previous similar messages [ 6399.161254] LustreError: 246492:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6399.173803] LustreError: 246492:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6399.334754] Lustre: server umount lustre-OST0000 complete [ 6403.042500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6403.047952] LustreError: Skipped 1 previous similar message [ 6408.639963] Lustre: server umount lustre-MDT0000 complete [ 6411.920838] LustreError: 245794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495987 with bad export cookie 2104606058336989495 [ 6411.922158] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.930049] LustreError: 245794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6412.249796] Lustre: server umount lustre-MDT0001 complete [ 6421.884724] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 6425.359454] Key type lgssc unregistered [ 6425.768414] LNet: 247429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6425.776415] LNetError: 247429:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6426.797374] LNet: Removed LNI 192.168.204.103@tcp [ 6427.744941] Key type .llcrypt unregistered [ 6427.751473] Key type ._llcrypt unregistered [ 6449.870212] Key type ._llcrypt registered [ 6449.872150] Key type .llcrypt registered [ 6449.955668] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6450.767140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6450.883078] alg: No test for adler32 (adler32-zlib) [ 6451.964106] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 6452.242760] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 6453.951243] Key type lgssc registered [ 6455.081949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6462.283089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6462.332384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.789291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.977462] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6474.225718] LustreError: 248854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6474.247235] LustreError: 248854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6476.460917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6476.893607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6481.011966] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6486.581425] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6490.426243] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6497.084872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6497.549990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6502.282209] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6507.465880] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6508.354478] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 6508.877986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6513.124760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6513.128054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.146642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6514.148455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6514.151349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.090213] LustreError: 251145:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6516.283582] Lustre: server umount lustre-OST0000 complete [ 6522.848321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6522.853928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.886946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6522.896032] Lustre: Skipped 1 previous similar message [ 6526.222024] LustreError: 251350:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6526.229554] LustreError: 251350:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6526.336118] Lustre: server umount lustre-MDT0000 complete [ 6529.505800] LustreError: 250013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6529.521405] LustreError: 250013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6529.791542] LustreError: 248834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496105 with bad export cookie 299129493599839552 [ 6529.795224] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6529.807860] LustreError: 248834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6529.963952] LustreError: 251552:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6529.968231] LustreError: 251552:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6530.244581] Lustre: server umount lustre-MDT0001 complete [ 6539.818861] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 6543.057383] Key type lgssc unregistered [ 6543.383882] LNet: 252085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6543.396879] LNetError: 252085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6543.432161] LNet: Removed LNI 192.168.204.103@tcp [ 6544.433160] Key type .llcrypt unregistered [ 6544.441979] Key type ._llcrypt unregistered [ 6558.175772] Key type ._llcrypt registered [ 6558.183472] Key type .llcrypt registered [ 6558.272523] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6559.301184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6559.360279] alg: No test for adler32 (adler32-zlib) [ 6560.448036] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 6560.692048] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 6562.399202] Key type lgssc registered [ 6563.913317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6575.834982] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6575.884251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6591.648732] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6596.768070] LustreError: 253238:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.103@tcp: failed processing log, type 4: rc = -110 [ 6623.519415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6628.976288] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6633.945080] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6635.388438] LustreError: 253902:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6635.508993] Lustre: server umount lustre-OST0000 complete [ 6647.952858] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 6650.546817] Key type lgssc unregistered [ 6650.891727] LNet: 254739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6650.903149] LNetError: 254739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6650.926596] LNet: Removed LNI 192.168.204.103@tcp [ 6651.681961] Key type .llcrypt unregistered [ 6651.693625] Key type ._llcrypt unregistered [ 6678.366783] Key type ._llcrypt registered [ 6678.368502] Key type .llcrypt registered [ 6678.448803] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6679.230913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6679.253674] alg: No test for adler32 (adler32-zlib) [ 6680.242906] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 6680.402762] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 6682.079173] Key type lgssc registered [ 6683.037361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6695.305858] loop0: detected capacity change from 0 to 204800000 [ 6695.545995] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6695.555365] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6695.998064] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6696.008856] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6696.045432] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6696.086352] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6696.122044] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6696.166782] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6696.267305] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6696.356263] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6697.471393] loop0: detected capacity change from 0 to 204800000 [ 6697.690755] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6698.259558] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6699.069887] loop0: detected capacity change from 0 to 204800000 [ 6699.603542] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6700.808403] loop0: detected capacity change from 0 to 204800000 [ 6701.043441] print_req_error: 11 callbacks suppressed [ 6701.043451] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6701.129866] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6701.293689] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6701.405484] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6702.250624] loop0: detected capacity change from 0 to 204800000 [ 6702.286971] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6707.018412] loop0: detected capacity change from 0 to 204800000 [ 6707.061551] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6710.872936] loop0: detected capacity change from 0 to 204800000 [ 6710.914605] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6714.378763] loop0: detected capacity change from 0 to 204800000 [ 6714.412216] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6717.703887] loop0: detected capacity change from 0 to 204800000 [ 6717.767287] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6717.813890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6717.828074] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.873155] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6717.879820] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6717.889142] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6717.897586] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6717.904108] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6717.912159] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6717.918737] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6717.926988] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6717.935970] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6717.942923] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6717.951802] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6717.960893] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6717.969676] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6717.980386] Lustre: 258125:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6722.982030] LustreError: 258392:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6723.091816] Lustre: server umount MGS complete [ 6723.807311] loop0: detected capacity change from 0 to 204800000 [ 6723.897858] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6724.226954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6724.272025] Lustre: lustre-MDT0000: Aborting client recovery [ 6725.113161] loop1: detected capacity change from 0 to 204800000 [ 6725.262533] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6725.359946] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6725.407930] Lustre: 258562:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6725.427678] Lustre: 258562:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6725.437904] Lustre: 258562:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6725.449414] Lustre: 258562:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6725.460619] Lustre: 258562:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6725.471903] Lustre: 258562:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6725.480290] Lustre: 258562:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6725.500657] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6725.653301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6725.689563] Lustre: lustre-MDT0001: Aborting client recovery [ 6726.513147] loop2: detected capacity change from 0 to 204800000 [ 6726.653815] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6726.742997] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6726.786919] Lustre: 258645:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6726.796123] Lustre: 258645:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6726.811507] Lustre: 258645:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6726.818889] Lustre: 258645:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6726.826099] Lustre: 258645:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6726.834963] Lustre: 258645:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6726.842440] Lustre: 258645:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6726.933911] Lustre: Found index 0 for lustre-OST0000, updating log [ 6727.017195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6727.040358] Lustre: lustre-OST0000: Aborting recovery [ 6727.715969] loop3: detected capacity change from 0 to 204800000 [ 6727.799433] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6727.844155] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6727.880760] Lustre: 258732:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6727.887643] Lustre: 258732:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6727.894400] Lustre: 258732:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6727.903874] Lustre: 258732:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6727.911775] Lustre: 258732:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6727.919716] Lustre: 258732:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6727.924986] Lustre: 258732:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6727.937648] Lustre: Found index 1 for lustre-OST0001, updating log [ 6728.032189] Lustre: lustre-OST0001: Aborting recovery [ 6733.317219] LustreError: 258504:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 6734.312359] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 6734.312973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 6734.357415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 6734.381561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 6735.335258] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 6735.340173] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 6739.725659] LustreError: 258472:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1820us: rc = -2 [ 6740.236113] LustreError: 258472:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 3288us: rc = -2 [ 6740.251149] LustreError: 258472:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 30 previous similar messages [ 6755.812837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6755.823513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6755.826430] Lustre: Skipped 3 previous similar messages [ 6755.840171] Lustre: Skipped 3 previous similar messages [ 6758.743195] LustreError: 259093:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6758.763720] LustreError: 259093:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6758.868696] Lustre: server umount lustre-MDT0000 complete [ 6759.512506] LustreError: 258454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496334 with bad export cookie 11463990555329533516 [ 6759.524784] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6760.927725] LustreError: 258985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.929177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.930216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6760.960637] LustreError: 258985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6760.970607] Lustre: Skipped 1 previous similar message [ 6764.959576] LustreError: 258985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.960929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6764.978055] LustreError: 258985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6764.981169] Lustre: Skipped 2 previous similar messages [ 6765.715604] LustreError: 259144:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6765.720025] LustreError: 259144:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6765.899995] Lustre: server umount lustre-MDT0001 complete [ 6772.533469] LustreError: 259195:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6772.539789] LustreError: 259195:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6772.657870] Lustre: server umount lustre-OST0000 complete [ 6779.385933] LustreError: 259246:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6779.388616] LustreError: 259246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6779.596389] Lustre: server umount lustre-OST0001 complete [ 6786.797534] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 01:19:21 (1762496361) [ 6809.437958] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 6816.151610] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6823.779273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6829.067969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6834.197454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6840.179391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6847.784801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6847.826898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.968366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6847.992653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6848.048489] Lustre: lustre-MDT0000: new disk, initializing [ 6848.112861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6848.123979] Lustre: Skipped 1 previous similar message [ 6848.138355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6852.309813] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6862.799345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6862.881352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6863.027152] Lustre: 262795:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6863.119256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6863.121783] Lustre: Skipped 1 previous similar message [ 6863.211841] Lustre: lustre-MDT0001: new disk, initializing [ 6863.366072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6863.434659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6863.450358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6867.124613] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6873.893881] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6877.293633] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6884.947506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6885.056773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6885.280161] Lustre: lustre-OST0000: new disk, initializing [ 6885.293316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6885.394139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6890.894529] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6895.154729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6895.157649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6895.242335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6897.588233] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6902.494159] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 6908.410139] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 6915.982424] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 6923.149129] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 6972.387950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.392845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.393494] LustreError: Skipped 1 previous similar message [ 6972.398555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.405198] Lustre: Skipped 1 previous similar message [ 6972.413785] Lustre: Skipped 1 previous similar message [ 6976.370109] LustreError: 266217:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6976.375746] LustreError: 266217:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6976.483357] Lustre: server umount lustre-OST0000 complete [ 6981.089371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6981.093846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.130336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6991.843517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6991.845996] Lustre: Skipped 4 previous similar messages [ 6994.400057] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6994.492047] LustreError: 266420:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6994.499840] LustreError: 266420:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6994.585412] Lustre: server umount lustre-MDT0000 complete [ 6996.964555] LustreError: 263936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.986521] LustreError: 263936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6997.573513] LustreError: 262788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496572 with bad export cookie 11463990555329536239 [ 6997.579694] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.603981] LustreError: 262788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6997.918239] Lustre: server umount lustre-MDT0001 complete [ 7007.123660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7011.787707] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7016.537447] Lustre: server umount MGS complete [ 7026.202145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.629878] LustreError: 268187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.654172] LustreError: 268199:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7026.714687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.579868] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7031.787398] LustreError: 268187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.977900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7041.394655] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7047.028597] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7050.519464] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7056.722278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7062.916687] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7068.359777] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7100.609149] LustreError: 270846:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7100.626043] LustreError: 270846:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7100.805677] Lustre: server umount lustre-OST0000 complete [ 7103.969762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7103.973571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.979286] Lustre: Skipped 1 previous similar message [ 7103.981902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7118.303146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7118.435971] Lustre: server umount lustre-MDT0000 complete [ 7118.819988] LustreError: 269193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.822658] Lustre: 256078:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762496678/real 1762496678] req@00000000f32a7763 x1848111273264256/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1762496694 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 7118.829771] LustreError: 269193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7118.852356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.858220] Lustre: Skipped 1 previous similar message [ 7121.705080] LustreError: 268167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496696 with bad export cookie 11463990555329538094 [ 7121.707624] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7121.729609] LustreError: 268167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7122.065585] Lustre: server umount lustre-MDT0001 complete [ 7130.224644] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 7132.509300] Key type lgssc unregistered [ 7132.710983] LNet: 271782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7132.714752] LNetError: 271782:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7133.735798] LNet: Removed LNI 192.168.204.103@tcp [ 7134.523286] Key type .llcrypt unregistered [ 7134.525881] Key type ._llcrypt unregistered [ 7166.517035] Key type ._llcrypt registered [ 7166.518959] Key type .llcrypt registered [ 7166.605195] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 7176.977989] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 7177.696170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7177.722314] alg: No test for adler32 (adler32-zlib) [ 7178.750506] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 7178.921802] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 7180.591184] Key type lgssc registered [ 7181.455922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7188.778255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7196.323336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7203.172859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7209.825590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7215.689082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7215.787659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7215.803723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.117979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7217.162364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7217.240858] Lustre: lustre-MDT0000: new disk, initializing [ 7217.335179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7217.349772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7221.210213] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7231.312146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7231.403250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7231.528367] Lustre: 275693:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7231.620090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7231.622863] Lustre: Skipped 1 previous similar message [ 7231.721888] Lustre: lustre-MDT0001: new disk, initializing [ 7231.850669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7231.915725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7231.927786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7236.171507] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7243.574499] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7247.500941] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7254.189818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7254.244648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7254.411090] Lustre: lustre-OST0000: new disk, initializing [ 7254.417051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7254.503272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7260.321355] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7262.282823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7262.288573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7262.422987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7267.760779] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7274.695766] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 7284.643657] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 7294.382857] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 7305.755047] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 7349.730547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.734501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7349.736297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7349.747048] Lustre: Skipped 1 previous similar message [ 7349.774962] LustreError: Skipped 1 previous similar message [ 7354.855937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7354.865330] Lustre: Skipped 2 previous similar messages [ 7355.478421] LustreError: 279333:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7355.629589] Lustre: server umount lustre-OST0000 complete [ 7359.968091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7359.971035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.997898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7365.099797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7365.102821] Lustre: Skipped 1 previous similar message [ 7370.210089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7374.305124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7374.389813] LustreError: 279536:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7374.397554] LustreError: 279536:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7374.573860] Lustre: server umount lustre-MDT0000 complete [ 7379.075787] LustreError: 275685:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496954 with bad export cookie 4680123629488561280 [ 7379.081097] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7379.167741] LustreError: 275685:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7379.484135] LustreError: 279738:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7379.488220] LustreError: 279738:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7379.789894] Lustre: server umount lustre-MDT0001 complete [ 7388.687777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.083261] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7396.837655] LustreError: 280681:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7396.843877] LustreError: 280681:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7396.891631] Lustre: server umount MGS complete [ 7405.034230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.444834] LustreError: 281304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.470353] LustreError: 281304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7405.477184] LustreError: 281316:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7405.535328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.554267] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7410.667062] LustreError: 281303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.755967] LustreError: 281303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7416.430180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7416.929099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7420.421798] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7425.467485] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7429.254591] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7435.734870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7436.033189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.602321] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7446.268584] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7481.316349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.322239] Lustre: Skipped 1 previous similar message [ 7481.331542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7481.334420] Lustre: Skipped 1 previous similar message [ 7481.335664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7484.665225] LustreError: 283967:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7484.678943] LustreError: 283967:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7484.872459] Lustre: server umount lustre-OST0000 complete [ 7488.481432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7488.491705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7488.500987] Lustre: Skipped 1 previous similar message [ 7500.774193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7500.777038] Lustre: Skipped 7 previous similar messages [ 7502.304922] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7502.371228] LustreError: 284169:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7502.374534] LustreError: 284169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7502.466816] Lustre: server umount lustre-MDT0000 complete [ 7505.892118] LustreError: 281304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.905722] LustreError: 281304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7506.170792] LustreError: 281283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497081 with bad export cookie 4680123629488563219 [ 7506.172076] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7506.178870] LustreError: 281283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7507.031942] Lustre: server umount lustre-MDT0001 complete [ 7516.785100] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 7518.942964] Key type lgssc unregistered [ 7519.189339] LNet: 284904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7519.194827] LNetError: 284904:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7519.203772] LNet: Removed LNI 192.168.204.103@tcp [ 7519.875451] Key type .llcrypt unregistered [ 7519.881749] Key type ._llcrypt unregistered [ 7536.748342] Key type ._llcrypt registered [ 7536.751509] Key type .llcrypt registered [ 7536.854615] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 7547.827285] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 7548.523289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7548.549755] alg: No test for adler32 (adler32-zlib) [ 7549.506756] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 7549.661247] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 7551.327340] Key type lgssc registered [ 7552.252355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7560.333765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7565.559748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7573.568354] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 7580.308992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7585.599667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7585.677573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7585.697911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.901045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7586.925420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7586.983887] Lustre: lustre-MDT0000: new disk, initializing [ 7587.104228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.138346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7590.616370] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7600.378155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7600.445988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7600.626283] Lustre: lustre-OST0000: new disk, initializing [ 7600.631461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7600.633598] Lustre: Skipped 1 previous similar message [ 7600.727304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7604.165310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7604.178677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7604.230795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7606.174938] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8372.193667] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8372.204350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8373.871375] LustreError: 289923:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8374.154893] Lustre: server umount lustre-MDT0000 complete [ 8377.251451] LustreError: 288091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497952 with bad export cookie 16512101410162738606 [ 8377.259941] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.292481] LustreError: 290126:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8377.297711] LustreError: 290126:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8377.428509] Lustre: server umount lustre-OST0000 complete [ 8401.646804] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 8409.037492] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 8417.132567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8422.819055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8428.268218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8433.715485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8440.166275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8440.279664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.640523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8440.674255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8440.789505] Lustre: lustre-MDT0000: new disk, initializing [ 8440.948511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8441.000919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8444.715508] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8453.723617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8453.779922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8453.842571] Lustre: 293170:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8453.891949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8453.894890] Lustre: Skipped 1 previous similar message [ 8453.939630] Lustre: lustre-MDT0001: new disk, initializing [ 8454.025913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8454.054805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8454.059605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8456.882927] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8462.847214] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8465.495258] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8470.446798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8470.513524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8470.710388] Lustre: lustre-OST0000: new disk, initializing [ 8470.716497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8470.788336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8474.444296] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8478.259738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8478.266548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8478.408253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8480.081414] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8483.190503] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8483.368891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8486.535472] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8486.705248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8488.929440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8488.930862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8488.937128] LustreError: Skipped 1 previous similar message [ 8488.949310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8494.057936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8494.064118] Lustre: Skipped 1 previous similar message [ 8494.339219] LustreError: 296110:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8494.343583] LustreError: 296110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8494.515757] Lustre: server umount lustre-OST0000 complete [ 8499.170899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.172075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.176653] Lustre: Skipped 2 previous similar messages [ 8499.179860] Lustre: Skipped 1 previous similar message [ 8504.086788] LustreError: 296311:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8504.103455] LustreError: 296311:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8504.208959] Lustre: server umount lustre-MDT0000 complete [ 8507.871891] LustreError: 294917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498083 with bad export cookie 16512101410165679453 [ 8507.876885] LustreError: 294917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8507.880572] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8508.198111] Lustre: server umount lustre-MDT0001 complete [ 8510.928952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 8512.533897] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 01:48:06 (1762498086) [ 8517.568714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.959603] LustreError: 297085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8517.984878] LustreError: 297085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8518.033684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.777477] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8523.237340] LustreError: 297084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8523.260629] LustreError: 297084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8528.356211] LustreError: 297084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.035792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8530.482216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8535.092591] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8540.907895] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8544.479553] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8550.661612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8551.056324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.584281] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8561.399729] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8567.378988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8567.440681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8567.564556] Lustre: lustre-OST0001: new disk, initializing [ 8567.573886] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8567.652477] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8572.699453] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8575.081577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8575.107076] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8575.160466] LustreError: 300039:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8576.225463] LustreError: 300035:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8576.230573] LustreError: 300035:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 8577.254355] LustreError: 300039:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8579.301362] LustreError: 300039:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8579.305256] LustreError: 300039:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 8581.673821] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8581.908214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8583.394136] LustreError: 300039:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8583.404802] LustreError: 300039:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 8586.451738] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa01e47b91000.ost_server_uuid 50 [ 8589.060894] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa01e47b91000.ost_server_uuid in FULL state after 0 sec [ 8591.655652] LustreError: 300039:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8591.660580] LustreError: 300039:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [ 8593.407588] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa01e47b91000.ost_server_uuid 50 [ 8595.148803] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa01e47b91000.ost_server_uuid in FULL state after 0 sec [ 8596.782610] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8611.313860] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8611.319762] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8611.328107] Lustre: Skipped 1 previous similar message [ 8611.337145] Lustre: Skipped 2 previous similar messages [ 8615.604303] LustreError: 300899:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8615.615683] LustreError: 300899:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8615.671129] Lustre: server umount lustre-OST0001 complete [ 8616.430584] LustreError: 300788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.451547] LustreError: 300788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8621.539282] LustreError: 300788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8621.541380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8621.574884] LustreError: 300788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8621.600329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8621.603157] Lustre: Skipped 1 previous similar message [ 8626.227131] LustreError: 301100:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8626.229280] LustreError: 301100:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8626.310970] Lustre: server umount lustre-OST0000 complete [ 8631.778645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8631.787545] Lustre: Skipped 2 previous similar messages [ 8635.012323] Lustre: server umount lustre-MDT0000 complete [ 8636.897402] LustreError: 297085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8636.909749] LustreError: 297085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8637.727754] LustreError: 297066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498212 with bad export cookie 16512101410165680335 [ 8637.730082] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8637.778687] LustreError: 297066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8638.173745] Lustre: server umount lustre-MDT0001 complete [ 8646.080847] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 8647.823355] Key type lgssc unregistered [ 8648.068192] LNet: 302036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8648.073260] LNetError: 302036:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8648.085325] LNet: Removed LNI 192.168.204.103@tcp [ 8648.667368] Key type .llcrypt unregistered [ 8648.668951] Key type ._llcrypt unregistered [ 8666.454856] Key type ._llcrypt registered [ 8666.456444] Key type .llcrypt registered [ 8666.526350] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 8667.293659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8667.312176] alg: No test for adler32 (adler32-zlib) [ 8668.379136] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 8668.579411] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 8670.247215] Key type lgssc registered [ 8671.184339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8677.301453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8677.330418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.832082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8682.317876] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8686.697957] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8689.125478] LustreError: 303417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8689.149585] LustreError: 303417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8691.846585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8692.126002] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8694.970715] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8701.264537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8701.502904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8702.507436] LustreError: 304972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.549327] LustreError: 304972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.554221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 8704.558020] LustreError: 304972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8705.126443] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8708.967984] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8709.605903] LustreError: 305068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8709.618636] LustreError: 305068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8713.731543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8713.911135] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8717.213136] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8719.174602] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8724.861502] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8729.158641] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8729.339578] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8829.921105] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8829.921824] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.930436] LustreError: Skipped 1 previous similar message [ 8829.941356] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8829.944500] Lustre: Skipped 1 previous similar message [ 8831.973467] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8831.979133] Lustre: Skipped 1 previous similar message [ 8837.092047] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8837.108553] Lustre: Skipped 1 previous similar message [ 8842.211253] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8842.220324] Lustre: Skipped 1 previous similar message [ 8843.233195] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8843.283189] LustreError: 306843:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8843.347271] Lustre: server umount lustre-OST0001 complete [ 8847.332329] LustreError: 304970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8847.359948] LustreError: 304970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8847.840436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8847.845951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8847.854970] Lustre: Skipped 1 previous similar message [ 8847.857840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8850.912547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8850.918917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8853.435288] LustreError: 307046:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8853.442585] LustreError: 307046:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8853.617791] Lustre: server umount lustre-OST0000 complete [ 8857.570342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8857.578072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8857.581463] Lustre: Skipped 4 previous similar messages [ 8871.391147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8871.469068] LustreError: 307247:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8871.478758] LustreError: 307247:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8871.534286] Lustre: server umount lustre-MDT0000 complete [ 8872.935624] LustreError: 306722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8872.946015] LustreError: 306722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8874.383311] LustreError: 303395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498449 with bad export cookie 11612879477840460448 [ 8874.388434] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.529809] LustreError: 307450:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8874.532270] LustreError: 307450:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8874.723107] Lustre: server umount lustre-MDT0001 complete [ 8884.132987] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 8886.795466] Key type lgssc unregistered [ 8887.177820] LNet: 307982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8887.191871] LNetError: 307982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8888.231766] LNet: Removed LNI 192.168.204.103@tcp [ 8889.229184] Key type .llcrypt unregistered [ 8889.231954] Key type ._llcrypt unregistered [ 8925.415346] Key type ._llcrypt registered [ 8925.416902] Key type .llcrypt registered [ 8925.491535] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 8926.178200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8926.232096] alg: No test for adler32 (adler32-zlib) [ 8927.163596] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 8927.323227] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 8928.983169] Key type lgssc registered [ 8929.892598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8938.812742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8938.837552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.185657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8943.628560] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8950.759272] LustreError: 310249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.774133] LustreError: 310249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8951.759836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8952.114815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8954.875529] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8956.773417] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8961.163336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8961.399782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.412821] LustreError: 311709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8965.849401] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8966.510081] LustreError: 311709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8966.523113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 8966.524453] LustreError: 311709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8971.750695] LustreError: 312108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8971.771545] LustreError: 312108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8971.885481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8972.056467] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8973.495115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 8976.116131] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8984.308631] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8993.229607] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9008.104872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9008.105842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9008.120708] Lustre: Skipped 1 previous similar message [ 9012.764790] LustreError: 313662:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9012.872650] Lustre: server umount lustre-MDT0000 complete [ 9013.218069] LustreError: 310981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9013.226141] LustreError: 310981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9015.773231] LustreError: 310230:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498591 with bad export cookie 13063178308020525346 [ 9015.777754] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9015.787179] LustreError: 310230:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9015.909482] LustreError: 313864:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 9015.912059] LustreError: 313864:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9016.107944] Lustre: server umount lustre-MDT0001 complete [ 9029.371660] LustreError: 314065:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9029.374856] LustreError: 314065:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9029.416264] Lustre: server umount lustre-OST0000 complete [ 9042.625677] LustreError: 314267:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9042.642457] LustreError: 314267:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9042.796040] Lustre: server umount lustre-OST0001 complete [ 9059.788641] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9068.907727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9069.220343] LustreError: 315446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9069.277698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9071.323127] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9077.504638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9077.776825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9081.343514] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9083.951961] Lustre: 316554:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9089.826858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9094.265367] LustreError: 316909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9094.290632] LustreError: 316909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9094.293913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 9095.202470] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9102.686661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9102.961422] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9102.970511] Lustre: Skipped 1 previous similar message [ 9104.412151] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 9108.757870] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9115.115297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9118.567788] Lustre: 318401:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9128.930021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9128.930829] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9128.936431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9128.936440] Lustre: Skipped 3 previous similar messages [ 9128.956737] Lustre: Skipped 5 previous similar messages [ 9132.506069] LustreError: 318505:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9132.510849] LustreError: 318505:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9132.630491] Lustre: server umount lustre-MDT0000 complete [ 9134.049287] LustreError: 315445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9134.075556] LustreError: 315445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9135.459991] LustreError: 315427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498710 with bad export cookie 13063178308020527047 [ 9135.463720] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9135.465364] LustreError: 315427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9135.788396] Lustre: server umount lustre-MDT0001 complete [ 9149.443112] LustreError: 318907:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9149.446046] LustreError: 318907:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 9149.512524] Lustre: server umount lustre-OST0000 complete [ 9163.366835] Lustre: server umount lustre-OST0001 complete [ 9166.913819] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 9168.662986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 9170.092708] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 01:59:04 (1762498744) [ 9191.968477] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9200.310205] loop0: detected capacity change from 0 to 1099511627776 [ 9200.998048] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 9201.013109] *** VALIDATE xfs *** [ 9201.030537] XFS (loop0): Mounting V5 Filesystem [ 9201.068866] XFS (loop0): Ending clean mount [ 9202.402299] loop1: detected capacity change from 0 to 18691697672192 [ 9251.549425] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9267.732978] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 9276.637924] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9285.586352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9292.704261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9299.773733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9307.438509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9314.316845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9314.370757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9314.654099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9314.705476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9314.827300] Lustre: lustre-MDT0000: new disk, initializing [ 9314.937861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9314.973890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9319.366346] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9330.531780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9330.667281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9330.816586] Lustre: 323514:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9330.965360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9330.967543] Lustre: Skipped 1 previous similar message [ 9331.163915] Lustre: lustre-MDT0001: new disk, initializing [ 9331.381023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9331.394910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9335.740992] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9342.950822] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9346.487490] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9353.476712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9353.579768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9353.798790] Lustre: lustre-OST0000: new disk, initializing [ 9353.803243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9355.732888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9355.754966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9355.904916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9359.642558] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9366.411919] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9369.705870] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9369.922763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9373.710251] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9373.885489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9376.225876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9376.230209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9376.232043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9376.232050] Lustre: Skipped 3 previous similar messages [ 9376.274905] Lustre: Skipped 1 previous similar message [ 9376.286080] LustreError: Skipped 1 previous similar message [ 9381.348717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9381.362324] Lustre: Skipped 2 previous similar messages [ 9381.505625] LustreError: 326454:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9381.524734] LustreError: 326454:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 9381.630979] Lustre: server umount lustre-OST0000 complete [ 9386.467877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9386.470034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9386.494572] Lustre: Skipped 1 previous similar message [ 9392.042856] Lustre: server umount lustre-MDT0000 complete [ 9395.319623] LustreError: 326656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498970 with bad export cookie 13063178308020528601 [ 9395.321695] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9395.334769] LustreError: 326656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9395.760195] Lustre: server umount lustre-MDT0001 complete [ 9405.063878] XFS (loop0): Unmounting Filesystem [ 9410.479202] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 02:03:04 (1762498984) [ 9417.258921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.700331] LustreError: 327677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9417.799170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9417.802178] Lustre: Skipped 2 previous similar messages [ 9422.175192] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9430.999259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9434.693607] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9440.197089] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9443.832793] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9450.173538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9456.082890] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9461.339639] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9478.115592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9478.124428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9478.126909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9478.134897] Lustre: Skipped 1 previous similar message [ 9478.154924] Lustre: Skipped 4 previous similar messages [ 9480.442169] LustreError: 330019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9480.456051] LustreError: 330019:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9480.651291] Lustre: server umount lustre-OST0000 complete [ 9487.841822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9487.849471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9490.598382] Lustre: server umount lustre-MDT0000 complete [ 9493.895937] LustreError: 327658:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762499069 with bad export cookie 13063178308020529483 [ 9493.902691] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9493.907283] LustreError: 327658:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9494.252360] Lustre: server umount lustre-MDT0001 complete [ 9504.370468] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 9506.652926] Key type lgssc unregistered [ 9506.946641] LNet: 330956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9506.962173] LNetError: 330956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9506.981652] LNet: Removed LNI 192.168.204.103@tcp [ 9507.771414] Key type .llcrypt unregistered [ 9507.775386] Key type ._llcrypt unregistered [ 9530.820585] Key type ._llcrypt registered [ 9530.824297] Key type .llcrypt registered [ 9530.895685] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9531.693220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9531.724796] alg: No test for adler32 (adler32-zlib) [ 9533.012275] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 9533.231439] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 9534.872261] Key type lgssc registered [ 9536.402846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9544.622695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9544.640979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9546.262082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.240058] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9556.456396] LustreError: 332484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9556.477652] LustreError: 332484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9558.430175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9558.872817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9562.002235] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9567.449758] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9570.660499] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9576.728625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9577.021783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.996991] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9588.916701] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9599.969122] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9599.980683] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.996138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9601.507876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9601.509794] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.888160] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [ 9604.891812] Lustre: Skipped 1 previous similar message [ 9609.997987] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [ 9610.001090] Lustre: Skipped 2 previous similar messages [ 9614.303188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9614.374309] LustreError: 334782:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9614.429021] Lustre: server umount lustre-MDT0001 complete [ 9615.113532] LustreError: 332480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9616.436276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9616.868703] LustreError: 334645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9616.879050] LustreError: 334645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9620.231659] LustreError: 333548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9622.112972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9622.216253] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9622.218714] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9622.265881] Lustre: 332474:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9622.501964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9625.565679] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9627.628379] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9631.212722] LustreError: 335773:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9631.219497] LustreError: 335773:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9631.224814] LustreError: 332476:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9631.292972] LustreError: 335782:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.103@tcp: cfg command failed: rc = -17 [ 9631.299476] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 9631.299476] [ 9932.590286] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [ 9932.600351] Lustre: Skipped 2 previous similar messages [ 9934.816434] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9934.822842] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9934.852303] Lustre: Skipped 1 previous similar message [ 9937.326549] LustreError: 344003:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9937.332893] LustreError: 344003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9937.470983] Lustre: server umount lustre-MDT0001 complete [ 9937.681485] LustreError: 333548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9937.704530] LustreError: 333548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9940.104319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9947.542994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9947.734248] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9947.736959] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9947.805689] Lustre: 332473:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9947.920481] LustreError: 332479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.953456] LustreError: 332479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9948.163964] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 9952.335911] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9953.263704] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9957.375670] LustreError: 344996:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9957.379213] LustreError: 344996:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9957.383220] LustreError: 332476:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9957.386210] LustreError: 332476:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9957.478757] LustreError: 345004:0:(obd_config.c:689:class_setup()) Device 8 already setup (type osp) [ 9957.484177] LustreError: 345004:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.103@tcp: cfg command failed: rc = -17 [ 9957.498327] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.103@tcp [ 9957.498327] [10260.247279] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [10260.261428] Lustre: Skipped 2 previous similar messages [10260.448306] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10260.451570] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.493500] Lustre: Skipped 1 previous similar message [10262.702723] LustreError: 353072:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10262.705857] LustreError: 353072:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10262.905985] Lustre: server umount lustre-MDT0001 complete [10265.361559] LustreError: 333548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10265.901282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10272.766285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10272.935875] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10272.946848] Lustre: Found index 1 for lustre-MDT0001, updating log [10273.020953] Lustre: 332473:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10273.413047] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10277.760636] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10278.916596] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10281.970397] LustreError: 354038:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [10281.990618] LustreError: 354038:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [10282.005330] LustreError: 332476:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [10282.018941] LustreError: 332476:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [10282.119095] LustreError: 354041:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [10282.130077] LustreError: 354041:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.103@tcp: cfg command failed: rc = -114 [10282.145523] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [10282.145523] [10599.397809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10599.405690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10599.412524] Lustre: Skipped 2 previous similar messages [10599.415050] Lustre: Skipped 3 previous similar messages [10613.216561] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10613.332837] LustreError: 362005:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10613.343626] LustreError: 362005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10613.440227] Lustre: server umount lustre-MDT0000 complete [10614.757222] LustreError: 332479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10614.789744] LustreError: 332479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10616.199192] LustreError: 344571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500191 with bad export cookie 16284303860091601466 [10616.202448] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10616.204971] LustreError: 344571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10619.874858] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.384795] LustreError: 362208:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10622.392687] LustreError: 362208:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10622.694407] Lustre: server umount lustre-MDT0001 complete [10626.214404] Lustre: server umount lustre-OST0000 complete [10635.901024] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [10646.219108] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [10656.082536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10662.672131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10668.118355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10674.742101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10682.131868] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 02:24:16 (1762500256) [10688.879725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10688.949510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.133939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10689.164669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10689.223504] Lustre: lustre-MDT0000: new disk, initializing [10689.377038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10689.397477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10693.381948] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10704.058210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10704.126195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10704.230870] Lustre: 364945:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10704.313958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10704.316397] Lustre: Skipped 1 previous similar message [10704.403192] Lustre: lustre-MDT0001: new disk, initializing [10704.515676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10704.584490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10704.604079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10708.452297] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10716.102678] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10720.146673] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10727.769735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10727.829248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10728.066967] Lustre: lustre-OST0000: new disk, initializing [10728.076181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10728.205213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10733.884545] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10735.653396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10735.672474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10735.767909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10741.380652] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10749.928034] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10749.941926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10749.960836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10749.963646] Lustre: Skipped 8 previous similar messages [10750.122566] LustreError: 367390:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10750.124820] LustreError: 367390:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10750.237654] Lustre: server umount lustre-MDT0000 complete [10751.456817] LustreError: 364957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10751.471596] LustreError: 364957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10753.963476] LustreError: 365740:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [10753.963876] LustreError: 365728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500329 with bad export cookie 16284303860091603440 [10753.968325] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10753.997555] LustreError: 365728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10754.346212] Lustre: server umount lustre-MDT0001 complete [10767.873486] LustreError: 367792:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10767.888302] LustreError: 367792:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10768.159405] Lustre: server umount lustre-OST0000 complete [10798.639193] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [10808.598377] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [10819.432642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10826.809696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10831.977604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10838.500192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10840.531339] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10574 sec ========= 02:26:54 (1762500414) [10842.415844] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:26:56 (1762500416) === [10846.018752] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:27:00 (1762500420) === [10887.646400] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [10890.909102] Key type lgssc unregistered [10891.293833] LNet: 372906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10891.301519] LNetError: 372906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10891.308789] LNet: Removed LNI 192.168.204.103@tcp [10892.172361] Key type .llcrypt unregistered [10892.174503] Key type ._llcrypt unregistered