[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 552860009 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002331] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.007483] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008095] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010175] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012068] SELinux: Initializing. [ 0.013054] *** VALIDATE selinux *** [ 0.020739] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024821] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027165] *** VALIDATE tmpfs *** [ 0.029101] *** VALIDATE proc *** [ 0.031195] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033432] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034202] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035026] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036045] Spectre V2 : User space: Vulnerable [ 0.037022] Speculative Store Bypass: Vulnerable [ 0.041172] debug: unmapping init [mem 0xffffffff84e59000-0xffffffff84e60fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044019] ... version: 2 [ 0.045008] ... bit width: 48 [ 0.046006] ... generic registers: 4 [ 0.047006] ... value mask: 0000ffffffffffff [ 0.048006] ... max period: 00007fffffffffff [ 0.049006] ... fixed-purpose events: 3 [ 0.050006] ... event mask: 000000070000000f [ 0.052250] rcu: Hierarchical SRCU implementation. [ 0.054499] smp: Bringing up secondary CPUs ... [ 0.055508] x86: Booting SMP configuration: [ 0.056015] .... node #0, CPUs: #1 #2 #3 [ 0.059293] smp: Brought up 1 node, 4 CPUs [ 0.061010] smpboot: Max logical packages: 1 [ 0.062008] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.197969] node 0 deferred pages initialised in 135ms [ 0.201111] devtmpfs: initialized [ 0.202275] x86/mm: Memory block size: 128MB [ 0.204989] gcov: version magic: 0x41383552 [ 0.208381] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.209176] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.211448] pinctrl core: initialized pinctrl subsystem [ 0.213163] [ 0.213592] ************************************************************* [ 0.215013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.217015] ** ** [ 0.219011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.221016] ** ** [ 0.223015] ** This means that this kernel is built to expose internal ** [ 0.225016] ** IOMMU data structures, which may compromise security on ** [ 0.227015] ** your system. ** [ 0.229008] ** ** [ 0.232013] ** If you see this message and you are not debugging the ** [ 0.234009] ** kernel, report this immediately to your vendor! ** [ 0.235008] ** ** [ 0.238010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239008] ************************************************************* [ 0.241815] NET: Registered protocol family 16 [ 0.243533] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.246059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.248055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.252050] cpuidle: using governor menu [ 0.253842] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.255493] PCI: Using configuration type 1 for base access [ 0.257142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.265276] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.270950] cryptd: max_cpu_qlen set to 1000 [ 0.274668] ACPI: Added _OSI(Module Device) [ 0.275000] ACPI: Added _OSI(Processor Device) [ 0.275012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.277017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280983] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.286585] ACPI: Interpreter enabled [ 0.287049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.289011] ACPI: Using IOAPIC for interrupt routing [ 0.291130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.292279] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.299000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.299000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.302015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.304075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.309326] acpiphp: Slot [2] registered [ 0.311084] acpiphp: Slot [3] registered [ 0.312062] acpiphp: Slot [4] registered [ 0.313104] acpiphp: Slot [5] registered [ 0.315090] acpiphp: Slot [6] registered [ 0.316077] acpiphp: Slot [7] registered [ 0.317088] acpiphp: Slot [8] registered [ 0.318130] acpiphp: Slot [9] registered [ 0.319118] acpiphp: Slot [10] registered [ 0.321083] acpiphp: Slot [11] registered [ 0.322068] acpiphp: Slot [12] registered [ 0.323056] acpiphp: Slot [13] registered [ 0.324074] acpiphp: Slot [14] registered [ 0.325120] acpiphp: Slot [15] registered [ 0.327081] acpiphp: Slot [16] registered [ 0.328089] acpiphp: Slot [17] registered [ 0.329070] acpiphp: Slot [18] registered [ 0.330123] acpiphp: Slot [19] registered [ 0.332108] acpiphp: Slot [20] registered [ 0.333131] acpiphp: Slot [21] registered [ 0.335120] acpiphp: Slot [22] registered [ 0.336110] acpiphp: Slot [23] registered [ 0.338143] acpiphp: Slot [24] registered [ 0.339076] acpiphp: Slot [25] registered [ 0.341061] acpiphp: Slot [26] registered [ 0.342066] acpiphp: Slot [27] registered [ 0.343088] acpiphp: Slot [28] registered [ 0.343936] acpiphp: Slot [29] registered [ 0.345044] acpiphp: Slot [30] registered [ 0.345891] acpiphp: Slot [31] registered [ 0.346053] PCI host bridge to bus 0000:00 [ 0.347022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.350024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.352026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.355020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.357008] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.359035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.361198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.364311] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368566] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.377015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.384059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.387022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.390026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.392016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.394048] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.397931] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.400042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402565] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420027] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.424028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.431746] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.438050] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.446036] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.486029] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.498805] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.513036] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.537035] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.558857] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.567039] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.577034] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.593026] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.604243] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.611014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.618021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.630019] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.642938] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.649031] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.655019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.667039] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.680000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.686022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.696028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.721019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.733374] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.736334] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.738345] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.740308] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.742369] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.749232] iommu: Default domain type: Passthrough [ 0.752333] SCSI subsystem initialized [ 0.754164] ACPI: bus type USB registered [ 0.755099] usbcore: registered new interface driver usbfs [ 0.757051] usbcore: registered new interface driver hub [ 0.758053] usbcore: registered new device driver usb [ 0.759145] pps_core: LinuxPPS API ver. 1 registered [ 0.760007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.762050] PTP clock support registered [ 0.763177] EDAC MC: Ver: 3.0.0 [ 0.765032] PCI: Using ACPI for IRQ routing [ 0.766805] NetLabel: Initializing [ 0.768011] NetLabel: domain hash size = 128 [ 0.769007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.770103] NetLabel: unlabeled traffic allowed by default [ 0.772123] vgaarb: loaded [ 0.774274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.775011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.778507] clocksource: Switched to clocksource kvm-clock [ 0.885244] VFS: Disk quotas dquot_6.6.0 [ 0.886170] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.888674] *** VALIDATE ramfs *** [ 0.889761] *** VALIDATE hugetlbfs *** [ 0.891388] pnp: PnP ACPI init [ 0.893562] pnp: PnP ACPI: found 6 devices [ 0.912044] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.916187] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.918797] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.921117] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.923609] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.926136] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.928910] NET: Registered protocol family 2 [ 0.931718] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.936850] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.940394] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.945060] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.948639] TCP: Hash tables configured (established 65536 bind 65536) [ 0.951222] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.953848] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956521] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959942] NET: Registered protocol family 1 [ 0.962808] RPC: Registered named UNIX socket transport module. [ 0.964631] RPC: Registered udp transport module. [ 0.966288] RPC: Registered tcp transport module. [ 0.967938] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.970856] NET: Registered protocol family 44 [ 0.972321] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.975057] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.978300] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.981691] PCI: CLS 0 bytes, default 64 [ 0.984814] Unpacking initramfs... [ 2.490897] debug: unmapping init [mem 0xffff9cf6fcc54000-0xffff9cf6fffbffff] [ 2.495463] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.497397] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.500168] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.068394] Initialise system trusted keyrings [ 3.069977] Key type blacklist registered [ 3.073127] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.085129] zbud: loaded [ 3.088382] *** VALIDATE nfs *** [ 3.089559] *** VALIDATE nfs4 *** [ 3.091077] pstore: using deflate compression [ 3.094699] Platform Keyring initialized [ 3.293881] NET: Registered protocol family 38 [ 3.296449] Key type asymmetric registered [ 3.298124] Asymmetric key parser 'x509' registered [ 3.300550] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.304343] io scheduler mq-deadline registered [ 3.306045] io scheduler kyber registered [ 3.307582] io scheduler bfq registered [ 3.313327] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.321245] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.329895] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.334170] ACPI: Power Button [PWRF] [ 3.447857] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.606342] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.935819] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.046826] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.356518] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.388646] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.424115] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.430608] Non-volatile memory driver v1.3 [ 4.432086] Linux agpgart interface v0.103 [ 4.462474] virtio_blk virtio1: [vda] 133800 512-byte logical blocks (68.5 MB/65.3 MiB) [ 4.466044] vda: detected capacity change from 0 to 68505600 [ 4.501668] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.508284] vdb: detected capacity change from 0 to 1073741824 [ 4.537743] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.544162] vdc: detected capacity change from 0 to 2621440000 [ 4.565547] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.572423] vdd: detected capacity change from 0 to 2621440000 [ 4.590551] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.595223] vde: detected capacity change from 0 to 4294967296 [ 4.621290] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.624090] vdf: detected capacity change from 0 to 4294967296 [ 4.635510] libphy: Fixed MDIO Bus: probed [ 4.648742] usbcore: registered new interface driver usbserial_generic [ 4.651192] usbserial: USB Serial support registered for generic [ 4.658321] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.663729] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.666221] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.669042] mousedev: PS/2 mouse device common for all mice [ 4.675886] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.680427] rtc_cmos 00:05: RTC can wake from S4 [ 4.693146] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.693462] rtc_cmos 00:05: registered as rtc0 [ 4.697694] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.700672] intel_pstate: CPU model not supported [ 4.705233] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.705452] hid: raw HID events driver (C) Jiri Kosina [ 4.710745] usbcore: registered new interface driver usbhid [ 4.712873] usbhid: USB HID core driver [ 4.714553] drop_monitor: Initializing network drop monitor service [ 4.716867] Initializing XFRM netlink socket [ 4.720800] NET: Registered protocol family 10 [ 4.725058] Segment Routing with IPv6 [ 4.726620] NET: Registered protocol family 17 [ 4.729612] mpls_gso: MPLS GSO support [ 4.735253] RAS: Correctable Errors collector initialized. [ 4.737311] AVX version of gcm_enc/dec engaged. [ 4.739180] AES CTR mode by8 optimization enabled [ 4.930373] sched_clock: Marking stable (4930346993, 0)->(5843515601, -913168608) [ 4.937229] registered taskstats version 1 [ 4.940877] Loading compiled-in X.509 certificates [ 4.944863] zswap: loaded using pool lzo/zbud [ 5.003506] Key type big_key registered [ 5.025568] Key type encrypted registered [ 5.027796] ima: No TPM chip found, activating TPM-bypass! [ 5.031171] ima: Allocated hash algorithm: sha1 [ 5.036262] ima: No architecture policies found [ 5.038228] evm: Initialising EVM extended attributes: [ 5.040360] evm: security.selinux [ 5.041604] evm: security.ima [ 5.042892] evm: security.capability [ 5.044736] evm: HMAC attrs: 0x1 [ 5.048640] rtc_cmos 00:05: setting system clock to 2025-10-07 04:55:18 UTC (1759812918) [ 5.060629] debug: unmapping init [mem 0xffffffff85e03000-0xffffffff85ffffff] [ 5.063822] debug: unmapping init [mem 0xffffffff84b82000-0xffffffff84e58fff] [ 5.077137] Write protecting the kernel read-only data: 28672k [ 5.081344] debug: unmapping init [mem 0xffffffff83203000-0xffffffff833fffff] [ 5.084218] debug: unmapping init [mem 0xffffffff83b14000-0xffffffff83bfffff] [ 5.134749] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.148425] systemd[1]: Detected virtualization kvm. [ 5.150262] systemd[1]: Detected architecture x86-64. [ 5.152233] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.180261] systemd[1]: No hostname configured. [ 5.181971] systemd[1]: Set hostname to . [ 5.184761] random: systemd: uninitialized urandom read (16 bytes read) [ 5.188239] systemd[1]: Initializing machine ID from random generator. [ 5.286429] random: ln: uninitialized urandom read (6 bytes read) [ 5.539073] random: systemd: uninitialized urandom read (16 bytes read) [ 5.543389] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.559454] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 5.593242] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.914254] device-mapper: uevent: version 1.0.3 [ 6.916720] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 8.124276] random: fast init done [ 8.315303] virtio_net virtio0 ens2: renamed from eth0 [ 9.189348] scsi host0: ata_piix [ 9.253753] scsi host1: ata_piix [ 9.255220] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.263521] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.559161] random: crng init done [ 13.560845] random: 7 urandom warning(s) missed due to ratelimiting [ 15.515370] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.418937] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.546371] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.165773] SELinux: Disabled at runtime. [ 19.256575] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.270582] systemd[1]: Detected virtualization kvm. [ 19.273041] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.288430] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.294862] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.307577] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.319814] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.329631] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.338431] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.346631] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 20.742423] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.490409] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.145761] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.414324] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.159757] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.229084] EDAC sbridge: Ver: 1.1.2 [ 26.261508] Key type dns_resolver registered [ 26.654430] NFS: Registering the id_resolver key type [ 26.656236] Key type id_resolver registered [ 26.658658] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg457-server login: [ 60.044143] libcfs: loading out-of-tree module taints kernel. [ 60.055117] Key type ._llcrypt registered [ 60.056450] Key type .llcrypt registered [ 60.099949] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 67.615275] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 68.093531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.101607] alg: No test for adler32 (adler32-zlib) [ 69.088406] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 69.405247] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 71.023166] Key type lgssc registered [ 71.505611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.503031] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 78.404423] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.784845] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.408240] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.952857] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.857346] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 96.397064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.422694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 96.432659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 97.518798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 97.532376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 97.573283] Lustre: lustre-MDT0000: new disk, initializing [ 97.597391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.603510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 98.904996] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.084508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.118560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 104.148713] Lustre: 6490:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 104.163279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 104.165881] Lustre: Skipped 1 previous similar message [ 104.203680] Lustre: lustre-MDT0001: new disk, initializing [ 104.227067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 104.236010] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 104.239585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 105.576123] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.866911] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 111.102909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.134528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 111.226683] Lustre: lustre-OST0000: new disk, initializing [ 111.228856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 111.245346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 112.171885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 112.175576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 112.199893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 113.169942] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 118.494776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.516848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 118.548104] Lustre: lustre-OST0001: new disk, initializing [ 118.549511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 118.564906] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 120.491505] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.876435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 120.879791] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 120.905277] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 126.402133] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 132.037718] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 138.880846] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing check_logdir /tmp/testlogs/ [ 140.256232] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing yml_node [ 141.628970] Lustre: DEBUG MARKER: Client: 2.16.58.105 [ 142.422334] Lustre: DEBUG MARKER: MDS: 2.16.58.105 [ 143.222562] Lustre: DEBUG MARKER: OSS: 2.16.58.105 [ 143.721598] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Oct 7 00:57:36 EDT 2025 [ 149.181588] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 149.722213] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 165.855707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 165.858639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.863393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 166.879833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 166.880018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 166.884455] Lustre: Skipped 1 previous similar message [ 166.886305] Lustre: Skipped 1 previous similar message [ 168.382059] LustreError: 12190:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 168.410924] Lustre: server umount lustre-MDT0000 complete [ 169.681832] LustreError: 6481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813083 with bad export cookie 16132945172998208443 [ 169.683191] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 169.686785] LustreError: 6481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 171.999750] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 171.999952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 172.006762] Lustre: Skipped 2 previous similar messages [ 172.008747] Lustre: Skipped 2 previous similar messages [ 175.007622] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 175.010137] Lustre: Skipped 2 previous similar messages [ 175.803933] LustreError: 12392:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 175.807514] LustreError: 12392:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 175.878478] Lustre: server umount lustre-MDT0001 complete [ 177.197073] LustreError: 12593:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 177.199641] LustreError: 12593:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 177.216466] Lustre: server umount lustre-OST0000 complete [ 178.495551] Lustre: server umount lustre-OST0001 complete [ 180.522181] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 183.436363] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 186.867613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.088120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.151863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 193.238148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.870243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.897045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.988536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.999402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 197.038395] Lustre: lustre-MDT0000: new disk, initializing [ 197.065596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.073844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 198.564794] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 202.789724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.811572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 202.845494] Lustre: 14972:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.950018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.236941] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 208.516448] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 211.802877] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 218.971526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 219.152183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 219.537576] Lustre: lustre-OST0000: new disk, initializing [ 219.540775] Lustre: Skipped 1 previous similar message [ 219.552206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.566664] Lustre: Skipped 2 previous similar messages [ 219.672408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.674633] Lustre: Skipped 1 previous similar message [ 221.254877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.258252] Lustre: Skipped 1 previous similar message [ 221.263024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.349859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.031775] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 235.379716] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 240.898376] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 241.237212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 246.454640] hrtimer: interrupt took 8618057 ns [ 246.666022] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 246.875260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 252.387125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 252.390766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.398757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 252.419145] Lustre: Skipped 1 previous similar message [ 255.047076] LustreError: 17913:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 255.057188] LustreError: 17913:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 255.858866] Lustre: server umount lustre-OST0000 complete [ 262.630739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.635206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.665320] Lustre: Skipped 1 previous similar message [ 262.677284] Lustre: Skipped 2 previous similar messages [ 266.353632] LustreError: 18114:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 266.372811] LustreError: 18114:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 266.580563] Lustre: server umount lustre-MDT0000 complete [ 267.748270] LustreError: 14984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 267.776863] LustreError: 14984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 271.349836] LustreError: 14963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813184 with bad export cookie 16132945172998209906 [ 271.353694] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.377239] LustreError: 14963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 272.036995] Lustre: server umount lustre-MDT0001 complete [ 278.043463] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 00:59:49 (1759813189) [ 284.385197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 284.853518] LustreError: 18842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.936451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 288.431232] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 290.275174] LustreError: 18842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.290231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 295.396372] LustreError: 18842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.425814] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 302.409729] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 305.622741] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 311.003304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 315.704320] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 319.808100] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 333.503579] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 339.938579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.942088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.944881] Lustre: Skipped 2 previous similar messages [ 339.949929] Lustre: Skipped 2 previous similar messages [ 343.771746] LustreError: 21163:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 343.775389] LustreError: 21163:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 343.847135] Lustre: server umount lustre-MDT0000 complete [ 345.059658] LustreError: 18841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.068739] LustreError: 18841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 346.260479] LustreError: 18823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813259 with bad export cookie 16132945172998210788 [ 346.263847] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 346.265770] LustreError: 18823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 346.517277] Lustre: server umount lustre-MDT0001 complete [ 359.761778] Lustre: server umount lustre-OST0000 complete [ 374.210695] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 381.466336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.789191] LustreError: 22909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.849082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.852543] Lustre: Skipped 2 previous similar messages [ 384.480168] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 390.493900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 393.336304] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 395.371191] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 400.009414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 403.898057] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 408.750655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 408.815189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 408.932862] Lustre: lustre-OST0001: new disk, initializing [ 408.936674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 410.084232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 410.092889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 410.187581] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 412.686854] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 419.085637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 420.999987] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 435.683776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.690329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 435.697882] Lustre: Skipped 2 previous similar messages [ 435.707405] Lustre: Skipped 3 previous similar messages [ 440.600081] LustreError: 26389:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 440.606440] LustreError: 26389:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 440.655930] Lustre: server umount lustre-MDT0000 complete [ 440.803084] LustreError: 22909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.813799] LustreError: 22909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 443.128034] LustreError: 22890:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813356 with bad export cookie 16132945172998211635 [ 443.129284] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.134494] LustreError: 22890:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 443.395772] Lustre: server umount lustre-MDT0001 complete [ 456.228622] Lustre: server umount lustre-OST0000 complete [ 469.367692] Lustre: server umount lustre-OST0001 complete [ 479.531985] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 486.812961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.106310] LustreError: 28167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.170353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.174033] Lustre: Skipped 3 previous similar messages [ 489.955466] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 495.715256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 498.254750] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 499.923578] Lustre: 29282:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 503.926617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 507.970705] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 513.780962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 518.035968] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 523.419938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 525.264619] Lustre: 31136:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 534.501895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.505136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 534.508484] Lustre: Skipped 3 previous similar messages [ 534.510861] Lustre: Skipped 1 previous similar message [ 538.898390] LustreError: 31339:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 538.905059] LustreError: 31339:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 538.957718] Lustre: server umount lustre-MDT0000 complete [ 540.791687] LustreError: 28146:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813454 with bad export cookie 16132945172998212930 [ 540.795279] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.797713] LustreError: 28146:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 541.018203] Lustre: server umount lustre-MDT0001 complete [ 552.969202] Lustre: server umount lustre-OST0000 complete [ 563.772657] Lustre: server umount lustre-OST0001 complete [ 567.292773] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 01:04:39 (1759813479) [ 579.929464] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 584.605530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.813829] LustreError: 33919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.824641] LustreError: 33919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 586.662429] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 590.677477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 592.553123] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 593.956760] Lustre: 35037:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 596.668067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 598.973857] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 602.259271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 604.589246] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 613.429047] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 614.883664] Lustre: 36890:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 625.633536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.640085] Lustre: Skipped 3 previous similar messages [ 628.039209] LustreError: 37043:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 628.041277] LustreError: 37043:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 628.076589] Lustre: server umount lustre-MDT0000 complete [ 629.617192] LustreError: 34656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813543 with bad export cookie 16132945172998214036 [ 629.619280] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 629.622726] LustreError: 34656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 629.803551] Lustre: server umount lustre-MDT0001 complete [ 640.650624] Lustre: server umount lustre-OST0000 complete [ 652.553627] Lustre: server umount lustre-OST0001 complete [ 656.094979] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 01:06:08 (1759813568) [ 662.724971] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 666.968027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.186962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.190081] Lustre: Skipped 7 previous similar messages [ 668.756261] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 672.350887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 674.093083] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 675.284948] Lustre: 40134:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 678.138205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 680.670839] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 684.222962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 686.572363] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 695.192387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 701.835827] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 701.838080] Lustre: Skipped 1 previous similar message [ 702.978298] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.57@tcp (stopping) [ 702.981290] Lustre: Skipped 6 previous similar messages [ 703.456975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.460056] LustreError: Skipped 1 previous similar message [ 709.150977] Lustre: server umount lustre-MDT0000 complete [ 710.423048] LustreError: 38997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813623 with bad export cookie 16132945172998215142 [ 710.426840] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.427916] LustreError: 38997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 713.191082] LustreError: 39011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.198360] LustreError: 39011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 716.627800] Lustre: server umount lustre-MDT0001 complete [ 719.829514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.199530] LustreError: 3636:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9cf7513b1180 x1845297657749888/t0(0) o250->MGC192.168.204.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 736.707521] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 739.718963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 740.903642] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 741.303599] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 743.216197] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 744.794937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 744.997751] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 745.000608] Lustre: Skipped 1 previous similar message [ 746.026248] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 753.776458] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 7 sec [ 755.170401] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.171164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 755.176248] Lustre: Skipped 10 previous similar messages [ 761.218452] LustreError: 44383:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 761.220305] LustreError: 44383:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 761.256116] Lustre: server umount lustre-MDT0000 complete [ 762.683445] LustreError: 38995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813676 with bad export cookie 16132945172998216318 [ 762.685789] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 762.687878] LustreError: 38995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 762.828426] Lustre: server umount lustre-MDT0001 complete [ 774.594496] Lustre: server umount lustre-OST0000 complete [ 786.162837] Lustre: server umount lustre-OST0001 complete [ 789.124353] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 01:08:21 (1759813701) [ 794.581491] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 798.035655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.518302] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 802.581374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 804.318860] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 805.475513] Lustre: 47474:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 805.479759] Lustre: 47474:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 807.981941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 810.292067] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 813.265036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 815.396298] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 829.068908] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 835.691779] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 843.747949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 846.163872] Lustre: server umount lustre-MDT0000 complete [ 847.489592] LustreError: 49354:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813760 with bad export cookie 16132945172998217102 [ 847.492957] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 847.494426] LustreError: 49354:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 847.657487] Lustre: server umount lustre-MDT0001 complete [ 859.330470] Lustre: server umount lustre-OST0000 complete [ 870.965973] Lustre: server umount lustre-OST0001 complete [ 874.179668] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 01:09:46 (1759813786) [ 875.023059] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 875.637325] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 01:09:48 (1759813788) [ 879.122696] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 882.082387] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 885.737969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 887.795942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 890.204284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 890.225706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.317044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 890.355086] Lustre: lustre-MDT0000: new disk, initializing [ 890.386306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 891.797559] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 894.939858] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.309180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 897.336204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 898.564190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 898.576687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 899.501363] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 902.637153] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 903.280147] Lustre: DEBUG MARKER: create test files [ 913.888371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 919.592849] Lustre: server umount lustre-OST0000 complete [ 921.169374] Lustre: server umount lustre-MDT0000 complete [ 925.046104] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 926.180963] Key type lgssc unregistered [ 926.328828] LNet: 54881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.332736] LNetError: 54881:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.342334] LNet: Removed LNI 192.168.204.157@tcp [ 926.699131] Key type .llcrypt unregistered [ 926.700877] Key type ._llcrypt unregistered [ 937.926739] Key type ._llcrypt registered [ 937.927995] Key type .llcrypt registered [ 937.972368] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 938.284821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 938.297027] alg: No test for adler32 (adler32-zlib) [ 939.172214] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 939.272140] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 940.863269] Key type lgssc registered [ 941.270612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 944.036168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 944.045937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.210499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.632254] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 948.515123] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 950.854876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 950.977404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.985237] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 954.788362] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 955.108908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 955.130422] Lustre: 57462:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 955.133863] Lustre: 57462:0:(osd_handler.c:2078:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 955.136065] Lustre: 57462:0:(osd_handler.c:2085:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 955.138831] Lustre: 57462:0:(osd_handler.c:2095:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 955.142408] Lustre: 57462:0:(osd_handler.c:2102:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 955.145410] Lustre: 57462:0:(osd_handler.c:2109:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 962.893670] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 964.729267] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 970.720621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.721276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.730924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 972.781149] LustreError: 58208:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 972.847801] Lustre: server umount lustre-OST0000 complete [ 974.223024] LustreError: 58411:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 974.225652] LustreError: 58411:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 974.310676] Lustre: server umount lustre-MDT0000 complete [ 978.043236] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 979.172536] Key type lgssc unregistered [ 979.316562] LNet: 58943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.319838] LNetError: 58943:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.332565] LNet: Removed LNI 192.168.204.157@tcp [ 979.682513] Key type .llcrypt unregistered [ 979.684120] Key type ._llcrypt unregistered [ 991.596820] Key type ._llcrypt registered [ 991.598295] Key type .llcrypt registered [ 991.636530] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 992.031688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.040439] alg: No test for adler32 (adler32-zlib) [ 992.889364] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 992.967438] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 994.543336] Key type lgssc registered [ 995.042561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 997.993148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 998.004120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.210737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.611279] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1002.421578] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1004.926397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1005.042268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.057601] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1009.033213] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1009.124413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1009.153113] Lustre: 61625:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1009.159564] Lustre: 61625:0:(osd_handler.c:2078:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1009.163456] Lustre: 61625:0:(osd_handler.c:2085:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1009.167552] Lustre: 61625:0:(osd_handler.c:2095:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1009.171532] Lustre: 61625:0:(osd_handler.c:2102:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1009.176120] Lustre: 61625:0:(osd_handler.c:2109:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1012.177703] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1016.814314] LustreError: 62318:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1016.871698] Lustre: server umount lustre-OST0000 complete [ 1027.555035] LustreError: 62519:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1027.558021] LustreError: 62519:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1027.633070] Lustre: server umount lustre-MDT0000 complete [ 1031.453821] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1032.557219] Key type lgssc unregistered [ 1032.683397] LNet: 63052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.685926] LNetError: 63052:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.697521] LNet: Removed LNI 192.168.204.157@tcp [ 1032.997939] Key type .llcrypt unregistered [ 1032.999215] Key type ._llcrypt unregistered [ 1039.075237] Key type ._llcrypt registered [ 1039.076479] Key type .llcrypt registered [ 1039.116595] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 1045.492248] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1045.898327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1045.911945] alg: No test for adler32 (adler32-zlib) [ 1046.770206] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1046.858344] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1048.439147] Key type lgssc registered [ 1048.798090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1051.986619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1054.485474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1056.717219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1058.914940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1061.162567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1061.184710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.189244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.267207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1062.280492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1062.326130] Lustre: lustre-MDT0000: new disk, initializing [ 1062.350976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.357143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1063.635601] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1067.925238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1067.951170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1067.988486] Lustre: 66152:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1068.023513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1068.025812] Lustre: Skipped 1 previous similar message [ 1068.066533] Lustre: lustre-MDT0001: new disk, initializing [ 1068.107128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.124661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1068.128773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1069.561476] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1072.801659] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1074.102097] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1076.423532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1076.451132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1076.547934] Lustre: lustre-OST0000: new disk, initializing [ 1076.549820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1076.579663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.431345] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1081.565749] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1081.850613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1081.854581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1081.885851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1082.922145] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1083.002392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1084.304600] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1084.380321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1086.944935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1086.946434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.947653] LustreError: Skipped 1 previous similar message [ 1086.951886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1086.954577] Lustre: Skipped 1 previous similar message [ 1091.055088] LustreError: 69091:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1091.128079] Lustre: server umount lustre-OST0000 complete [ 1093.599916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1093.602845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.606873] Lustre: Skipped 1 previous similar message [ 1093.609333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.184168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.185087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.489910] LustreError: 69292:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1098.492622] LustreError: 69292:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1098.518835] Lustre: server umount lustre-MDT0000 complete [ 1099.735342] LustreError: 66145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814013 with bad export cookie 9176604520046809630 [ 1099.739635] LustreError: 66145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1099.740180] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.809299] LustreError: 69493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1099.811914] LustreError: 69493:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1099.891468] Lustre: server umount lustre-MDT0001 complete [ 1102.768049] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 01:13:35 (1759814015) [ 1106.109261] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1110.264989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1118.355341] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 1121.628430] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1125.018404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1127.425800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1129.471231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1131.465515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1133.722732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1133.747787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.832830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1133.843166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1133.877793] Lustre: lustre-MDT0000: new disk, initializing [ 1133.901919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.909084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1135.191736] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1139.236384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1139.261398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1139.293334] Lustre: 73721:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1139.316727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1139.319073] Lustre: Skipped 1 previous similar message [ 1139.356435] Lustre: lustre-MDT0001: new disk, initializing [ 1139.390339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1139.401958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1139.405024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1140.647102] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1143.758631] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.916281] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1147.104466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1147.125123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1149.038420] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1152.045720] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1152.500749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1152.504921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1152.538590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1153.309272] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1153.382205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1154.631787] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1154.704795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1157.600601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.602789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.603747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.606592] Lustre: Skipped 1 previous similar message [ 1157.608354] Lustre: Skipped 1 previous similar message [ 1161.387072] LustreError: 76657:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1161.389020] LustreError: 76657:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1161.445078] Lustre: server umount lustre-OST0000 complete [ 1162.720058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.720671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.723925] Lustre: Skipped 1 previous similar message [ 1162.727628] Lustre: Skipped 2 previous similar messages [ 1168.702518] LustreError: 76859:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1168.704583] LustreError: 76859:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1168.732698] Lustre: server umount lustre-MDT0000 complete [ 1169.892297] LustreError: 73713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814083 with bad export cookie 9176604520046810512 [ 1169.895725] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.895922] LustreError: 73713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1170.023135] Lustre: server umount lustre-MDT0001 complete [ 1172.542644] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 01:14:45 (1759814085) [ 1174.551183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.680510] LustreError: 77730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.685677] LustreError: 77730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1174.699316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.701098] Lustre: Skipped 1 previous similar message [ 1175.808918] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1178.493704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.987340] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1181.761355] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1182.997207] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1185.262253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1187.347874] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1189.298395] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1190.507927] Lustre: 80061:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting 207cc312-8f8a-4325-a33b-643c808bc9da at adminstrative request [ 1191.903861] LustreError: MGC192.168.204.157@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 1191.906486] LustreError: Skipped 1 previous similar message [ 1191.907970] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1191.911434] Lustre: 77721:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -107 [ 1191.911568] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1202.144362] Lustre: MGS: Client 207cc312-8f8a-4325-a33b-643c808bc9da (at 0@lo) reconnecting [ 1202.147508] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x7f59d930f5a258f4 to 0x7f59d930f5a25b6a [ 1202.150747] Lustre: MGC192.168.204.157@tcp: Connection restored to 0@lo (at 0@lo) [ 1223.090514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1223.122655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1223.180581] Lustre: lustre-OST0001: new disk, initializing [ 1223.182115] Lustre: Skipped 1 previous similar message [ 1223.184930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1223.188652] Lustre: Skipped 1 previous similar message [ 1223.222710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.225192] Lustre: Skipped 2 previous similar messages [ 1225.197570] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1228.373399] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1229.397169] LustreError: 81191:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1229.399100] LustreError: 81191:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1229.414827] Lustre: server umount lustre-OST0001 complete [ 1230.816422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1230.816868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.819810] LustreError: Skipped 1 previous similar message [ 1230.820703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.825481] Lustre: Skipped 1 previous similar message [ 1230.827041] Lustre: Skipped 2 previous similar messages [ 1230.834816] LustreError: 79326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.939102] LustreError: 79887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.943432] LustreError: 79887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1241.061431] LustreError: 79326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.066868] LustreError: 79326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1245.151200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1245.226145] Lustre: server umount lustre-OST0000 complete [ 1250.272097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1250.276021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.282828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.285930] Lustre: Skipped 4 previous similar messages [ 1252.604217] LustreError: 81602:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 1252.606745] LustreError: 81602:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1252.635966] Lustre: server umount lustre-MDT0000 complete [ 1253.945937] LustreError: 80460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814167 with bad export cookie 9176604520046812010 [ 1253.951151] LustreError: 80460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1253.951751] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.110900] Lustre: server umount lustre-MDT0001 complete [ 1256.898285] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 01:16:09 (1759814169) [ 1257.383518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1257.914760] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 01:16:10 (1759814170) [ 1258.417926] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1258.963511] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 01:16:11 (1759814171) [ 1259.431856] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1259.965903] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 01:16:12 (1759814172) [ 1260.468043] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1262.076275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1262.404052] loop: module loaded [ 1262.408839] loop0: detected capacity change from 0 to 4294967296 [ 1262.419507] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1262.904808] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1263.876119] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1265.198471] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1267.996129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1269.281486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1270.497686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1271.741793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1273.841574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1273.866938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1273.877094] LustreError: 84371:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1273.880520] LustreError: 84371:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1273.884297] LustreError: 84371:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1273.888369] LustreError: 84371:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1279.557261] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 1282.417449] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1285.632150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1287.657542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1289.528687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1291.458259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1293.590499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1293.610459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.683755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1293.695372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1293.728442] Lustre: lustre-MDT0000: new disk, initializing [ 1293.750858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.757309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1294.980477] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1298.941427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1298.961924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1298.990105] Lustre: 87209:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1299.083630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1300.312628] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1303.343731] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1304.471237] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1306.579590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1306.603635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1308.487715] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1311.474455] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1312.713178] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1314.293777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1314.326814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1314.828889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1316.161065] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1316.240770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1319.393441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1319.393983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.396244] LustreError: Skipped 1 previous similar message [ 1319.401659] Lustre: Skipped 1 previous similar message [ 1319.403587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1319.406517] Lustre: Skipped 3 previous similar messages [ 1322.924137] LustreError: 90162:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1322.925831] LustreError: 90162:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1322.984421] Lustre: server umount lustre-OST0000 complete [ 1330.262134] Lustre: server umount lustre-MDT0000 complete [ 1331.436025] LustreError: 87200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814244 with bad export cookie 9176604520046812577 [ 1331.438572] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1331.440457] LustreError: 87200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1331.570974] Lustre: server umount lustre-MDT0001 complete [ 1334.211093] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 01:17:26 (1759814246) [ 1336.393081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.534200] LustreError: 91237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.791785] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1340.488778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1341.874986] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1343.573566] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1344.714081] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1346.842752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1348.668943] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1350.319418] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1351.600691] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1361.878374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 1363.144940] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1363.220668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1365.368033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1365.393508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1365.444722] Lustre: lustre-OST0001: new disk, initializing [ 1365.446948] Lustre: Skipped 2 previous similar messages [ 1365.449260] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1365.451953] Lustre: Skipped 3 previous similar messages [ 1365.482346] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1365.484416] Lustre: Skipped 5 previous similar messages [ 1367.302578] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1370.304216] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1371.577292] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1374.204847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1374.208494] Lustre: Skipped 2 previous similar messages [ 1374.210710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1374.233123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1374.704751] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1375.895514] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1375.969979] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1385.441166] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1389.988251] Lustre: Failing over lustre-MDT0000 [ 1390.159583] LustreError: 96208:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1390.162389] LustreError: 96208:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1390.194177] Lustre: server umount lustre-MDT0000 complete [ 1392.096164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1392.100241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.104223] Lustre: Skipped 2 previous similar messages [ 1392.105987] LustreError: 91236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.761512] LDISKFS-fs (dm-0): recovery complete [ 1404.763156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.798185] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.905370] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1406.170275] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1407.976313] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1407.977298] Lustre: 96327:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1410.016845] Lustre: 91255:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1410.020678] Lustre: 91255:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 1410.025289] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1410.039143] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1410.351069] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1410.664813] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1410.668547] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 1410.983045] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1410.986203] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 1411.305423] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1411.309389] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 8 previous similar messages [ 1411.936740] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1411.939704] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 3 previous similar messages [ 1412.255072] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1412.256993] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 3 previous similar messages [ 1413.511939] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1413.516838] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 6 previous similar messages [ 1414.143715] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1414.146831] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 1414.463159] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1414.466223] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 1417.593290] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1417.596527] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 12 previous similar messages [ 1418.225847] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1418.230978] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 1418.543158] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1418.546767] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 1425.729028] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1425.732746] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 25 previous similar messages [ 1426.361132] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1426.364606] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 25 previous similar messages [ 1426.679153] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1426.683247] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 25 previous similar messages [ 1441.977299] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1441.981601] Lustre: 96955:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 51 previous similar messages [ 1442.609664] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1442.613012] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 51 previous similar messages [ 1442.927121] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1442.929637] LustreError: 96955:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 51 previous similar messages [ 1467.279207] Lustre: 96955:0:(ldlm_lib.c:2375:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1467.282753] Lustre: 96955:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1467.285946] Lustre: 96955:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client cc4824b5-6cee-46e1-bf0a-d9db5b5b0c24@192.168.204.57@tcp [ 1467.291383] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1467.294262] Lustre: 96955:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@000000001e44e3e1 x1845298675649408/t0(8589934780) o35->cc4824b5-6cee-46e1-bf0a-d9db5b5b0c24@192.168.204.57@tcp:746/0 lens 392/0 e 0 to 0 dl 1759814391 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1467.303950] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1467.307495] Lustre: lustre-MDT0000: Denying connection for new client cc4824b5-6cee-46e1-bf0a-d9db5b5b0c24 (at 192.168.204.57@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1467.310297] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1467.324903] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1467.325169] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1467.327997] Lustre: Skipped 2 previous similar messages [ 1467.345264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 1467.345340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 1472.992417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1472.997107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1472.999445] Lustre: Skipped 4 previous similar messages [ 1474.235903] Lustre: server umount lustre-OST0000 complete [ 1476.578413] LustreError: 95534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1476.583882] LustreError: 95534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 1481.579578] Lustre: server umount lustre-OST0001 complete [ 1488.983454] Lustre: server umount lustre-MDT0000 complete [ 1490.224198] LustreError: 91218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814403 with bad export cookie 9176604520046863761 [ 1490.227042] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.228984] LustreError: 91218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1490.367631] Lustre: server umount lustre-MDT0001 complete [ 1493.088736] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 01:20:05 (1759814405) [ 1495.639364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1495.642948] Lustre: *** cfs_fail_loc=197, val=0*** [ 1511.071740] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1516.191475] LustreError: 98916:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 1521.318545] Lustre: *** cfs_fail_loc=197, val=0*** [ 1521.319855] Lustre: Skipped 65 previous similar messages [ 1541.791319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.793630] Lustre: Skipped 1 previous similar message [ 1543.717092] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1545.539099] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1546.183349] LustreError: 99579:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1546.184961] LustreError: 99579:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1546.247254] Lustre: server umount lustre-OST0000 complete [ 1548.956684] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 01:21:01 (1759814461) [ 1562.652135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.801350] LustreError: 100423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.124594] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1567.235876] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1569.191019] LustreError: 100454:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1569.193971] LustreError: 100454:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1569.196689] LustreError: 100454:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1569.340901] Lustre: server umount lustre-MDT0000 complete [ 1574.872518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1581.043676] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 1584.576290] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1588.030517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1590.234919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1592.266553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1594.438308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1596.763606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1596.788326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.870157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1596.883353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1596.919320] Lustre: lustre-MDT0000: new disk, initializing [ 1596.949888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1598.357042] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1602.459151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1602.481995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1602.508784] Lustre: 104488:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1602.603927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1603.862016] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1606.906044] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1608.078187] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1610.281100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1610.304839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1612.306670] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1615.248622] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1615.897896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1616.479721] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1616.550927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1617.757975] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1617.841775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1620.962375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1620.962584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.965345] LustreError: Skipped 2 previous similar messages [ 1620.970754] Lustre: Skipped 9 previous similar messages [ 1620.973471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1620.975416] Lustre: Skipped 6 previous similar messages [ 1624.621052] Lustre: server umount lustre-OST0000 complete [ 1632.020217] Lustre: server umount lustre-MDT0000 complete [ 1633.229345] LustreError: 104479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814546 with bad export cookie 9176604520046874429 [ 1633.232651] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1633.233413] LustreError: 104479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1633.379683] Lustre: server umount lustre-MDT0001 complete [ 1636.036666] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 01:22:28 (1759814548) [ 1638.850319] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1639.881438] Key type lgssc unregistered [ 1640.000525] LNet: 108554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.003289] LNetError: 108554:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.013460] LNet: Removed LNI 192.168.204.157@tcp [ 1640.313325] Key type .llcrypt unregistered [ 1640.314373] Key type ._llcrypt unregistered [ 1645.915867] Key type ._llcrypt registered [ 1645.916803] Key type .llcrypt registered [ 1645.952345] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 1652.055337] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1652.403432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1652.452026] alg: No test for adler32 (adler32-zlib) [ 1653.311283] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1653.404024] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1654.991192] Key type lgssc registered [ 1655.402424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1658.266165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1660.460588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1662.461183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1664.483528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1666.358590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1668.580488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1668.600896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1668.606866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.683480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1669.694308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1669.729845] Lustre: lustre-MDT0000: new disk, initializing [ 1669.751183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.757203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1671.040352] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1675.155345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1675.179424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1675.267705] Lustre: lustre-OST0000: new disk, initializing [ 1675.270282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1675.271923] Lustre: Skipped 1 previous similar message [ 1675.293184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.713447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1675.717184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1675.724706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1677.187985] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1681.330685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1681.354137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1681.405899] Lustre: lustre-OST0001: new disk, initializing [ 1681.408437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1681.434424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1681.576893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1681.580536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1681.589813] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1683.279870] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1692.129065] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.129899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1692.134578] Lustre: Skipped 1 previous similar message [ 1693.248271] LustreError: 114106:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1693.332920] Lustre: server umount lustre-MDT0000 complete [ 1695.603656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1697.067376] LustreError: 111862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814610 with bad export cookie 8758761618618133142 [ 1697.069133] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.070733] LustreError: 111862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1697.092734] LustreError: 114699:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1697.094835] LustreError: 114699:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1697.109895] Lustre: server umount lustre-OST0000 complete [ 1698.301918] LustreError: 114899:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1698.303942] LustreError: 114899:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1698.360807] Lustre: server umount lustre-OST0001 complete [ 1701.264049] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 01:23:33 (1759814613) [ 1704.338876] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1707.346916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1707.689088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1709.439761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1709.766043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1711.860091] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 01:23:44 (1759814624) [ 1716.099680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1716.788893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1717.464263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1718.121176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1718.464464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1719.139884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1724.740106] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 1727.798626] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1731.128754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1733.367148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1735.438068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1737.581470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1739.869233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1739.885530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.965260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1739.976143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1740.015730] Lustre: lustre-MDT0000: new disk, initializing [ 1740.044076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.052356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1741.350064] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1745.548334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1745.576246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1745.611668] Lustre: 120219:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1745.637636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1745.639488] Lustre: Skipped 1 previous similar message [ 1745.674579] Lustre: lustre-MDT0001: new disk, initializing [ 1745.711425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1745.724506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1745.729559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1747.155608] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1750.249098] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1751.452381] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1753.703645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1753.727668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1753.832269] Lustre: lustre-OST0000: new disk, initializing [ 1753.834096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1753.873219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.124469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1755.128205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1755.161528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1755.804266] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1758.882704] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1760.167067] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1760.239936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1761.508864] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1761.579854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1765.345146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1765.346041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.348374] LustreError: Skipped 1 previous similar message [ 1765.354551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.357433] Lustre: Skipped 2 previous similar messages [ 1768.300108] LustreError: 123158:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1768.303027] LustreError: 123158:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1768.369672] Lustre: server umount lustre-OST0000 complete [ 1770.464479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.465050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.468594] Lustre: Skipped 2 previous similar messages [ 1775.583997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.585747] Lustre: Skipped 2 previous similar messages [ 1775.739170] LustreError: 123359:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1775.741590] LustreError: 123359:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1775.773060] Lustre: server umount lustre-MDT0000 complete [ 1777.011132] LustreError: 120209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814690 with bad export cookie 8758761618618134353 [ 1777.013155] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1777.016062] LustreError: 120209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1777.177402] Lustre: server umount lustre-MDT0001 complete [ 1779.799416] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 01:24:52 (1759814692) [ 1781.999443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.178713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.475175] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1786.378489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1787.912733] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1789.692411] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1790.862417] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1793.165518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1795.180905] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1796.928647] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1809.277797] Lustre: *** cfs_fail_loc=159, val=0*** [ 1809.279442] Lustre: Skipped 1 previous similar message [ 1825.780992] Lustre: lustre-MDT0000: Client ad3e5fa6-96a1-4a20-86cf-8275ca4072f4 (at 192.168.204.57@tcp) reconnecting [ 1826.278537] Lustre: *** cfs_fail_loc=159, val=0*** [ 1826.280021] Lustre: Skipped 3 previous similar messages [ 1842.675250] Lustre: lustre-MDT0000: Client ad3e5fa6-96a1-4a20-86cf-8275ca4072f4 (at 192.168.204.57@tcp) reconnecting [ 1847.777308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1847.780539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.782168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1847.785693] Lustre: Skipped 1 previous similar message [ 1849.900237] LustreError: 126735:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1849.903209] LustreError: 126735:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1849.976888] Lustre: server umount lustre-OST0000 complete [ 1852.896789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.900385] Lustre: Skipped 1 previous similar message [ 1857.365459] Lustre: server umount lustre-MDT0000 complete [ 1858.015802] LustreError: 124231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.022272] LustreError: 124231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1858.599264] LustreError: 126939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814772 with bad export cookie 8758761618618135221 [ 1858.602676] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.604036] LustreError: 126939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1858.735744] Lustre: server umount lustre-MDT0001 complete [ 1862.237777] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1863.279783] Key type lgssc unregistered [ 1863.415311] LNet: 127672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.418287] LNetError: 127672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.424288] LNet: Removed LNI 192.168.204.157@tcp [ 1863.736890] Key type .llcrypt unregistered [ 1863.737935] Key type ._llcrypt unregistered [ 1872.121834] Key type ._llcrypt registered [ 1872.122958] Key type .llcrypt registered [ 1872.157418] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1872.561861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.571495] alg: No test for adler32 (adler32-zlib) [ 1873.424729] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1873.503303] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1875.079157] Key type lgssc registered [ 1875.422015] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.896262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1877.902812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.042067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.311214] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1882.989402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1883.103361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.295248] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1885.940838] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.006301] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1889.081099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1889.185158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.212829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 1890.940662] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1892.585664] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1898.001157] Lustre: *** cfs_fail_loc=159, val=0*** [ 1914.355958] Lustre: lustre-MDT0000: Client 40cf76d0-9cdb-496e-a36a-13db2edf1f9f (at 192.168.204.57@tcp) reconnecting [ 1915.052679] Lustre: *** cfs_fail_loc=159, val=0*** [ 1915.054711] Lustre: Skipped 1 previous similar message [ 1931.250487] Lustre: lustre-MDT0001: Client 40cf76d0-9cdb-496e-a36a-13db2edf1f9f (at 192.168.204.57@tcp) reconnecting [ 1931.725767] Lustre: *** cfs_fail_loc=159, val=0*** [ 1947.122328] Lustre: lustre-MDT0001: Client 40cf76d0-9cdb-496e-a36a-13db2edf1f9f (at 192.168.204.57@tcp) reconnecting [ 1948.321344] Lustre: 131746:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1949.171752] Lustre: *** cfs_fail_loc=159, val=0*** [ 1949.172924] Lustre: Skipped 4 previous similar messages [ 1965.557255] Lustre: lustre-MDT0000: Client f3376a58-e7c4-474b-a2ec-4df3f4182edb (at 192.168.204.57@tcp) reconnecting [ 1966.136376] Lustre: *** cfs_fail_loc=159, val=0*** [ 1966.138231] Lustre: Skipped 15 previous similar messages [ 1982.452400] Lustre: lustre-MDT0000: Client f3376a58-e7c4-474b-a2ec-4df3f4182edb (at 192.168.204.57@tcp) reconnecting [ 1982.909591] Lustre: 132037:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1985.506849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1985.507140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1985.509762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.514300] Lustre: Skipped 1 previous similar message [ 1985.518078] Lustre: Skipped 1 previous similar message [ 1990.122082] LustreError: 132134:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1990.179511] Lustre: server umount lustre-OST0000 complete [ 1995.744757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.745299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1995.748434] Lustre: Skipped 1 previous similar message [ 1995.752024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1995.754640] Lustre: Skipped 1 previous similar message [ 1997.498700] LustreError: 132335:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1997.501744] LustreError: 132335:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1997.528699] Lustre: server umount lustre-MDT0000 complete [ 1998.740316] LustreError: 131865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814912 with bad export cookie 5117277691916800921 [ 1998.744561] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1998.745144] LustreError: 131865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1998.807375] LustreError: 132536:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1998.809688] LustreError: 132536:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1998.885698] Lustre: server umount lustre-MDT0001 complete [ 2002.516294] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2003.548778] Key type lgssc unregistered [ 2003.673358] LNet: 133067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.675484] LNetError: 133067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.683287] LNet: Removed LNI 192.168.204.157@tcp [ 2003.994521] Key type .llcrypt unregistered [ 2003.995530] Key type ._llcrypt unregistered [ 2012.675023] Key type ._llcrypt registered [ 2012.676759] Key type .llcrypt registered [ 2012.714973] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2013.023552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2013.057910] alg: No test for adler32 (adler32-zlib) [ 2013.928087] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2014.019329] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2015.599131] Key type lgssc registered [ 2015.960662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2018.637140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2018.643728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.793374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.024783] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2023.923469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2024.048471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2025.378638] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2027.349556] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2028.650145] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2031.015369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2031.132096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.099867] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2034.216538] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2034.834700] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2046.503589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2052.831759] Lustre: 136711:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2053.531325] Lustre: 136759:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2054.044349] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2055.132701] Lustre: 136905:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2055.475199] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2055.477430] Lustre: Skipped 1 previous similar message [ 2055.791832] Lustre: 137003:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2058.797104] LustreError: 137101:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2058.863250] Lustre: server umount lustre-OST0000 complete [ 2061.793180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.794102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.799929] Lustre: Skipped 1 previous similar message [ 2061.804584] Lustre: Skipped 1 previous similar message [ 2066.912233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2066.914694] Lustre: Skipped 1 previous similar message [ 2069.986151] LustreError: 137303:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2069.988672] LustreError: 137303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2070.012898] Lustre: server umount lustre-MDT0000 complete [ 2071.328184] LustreError: 134419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814984 with bad export cookie 408265958173666718 [ 2071.331653] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2071.333309] LustreError: 134419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2080.738776] LustreError: 137508:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2080.740978] LustreError: 137508:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2080.825163] Lustre: server umount lustre-MDT0001 complete [ 2084.493812] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2085.668760] Key type lgssc unregistered [ 2085.806421] LNet: 138040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.809862] LNetError: 138040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.820601] LNet: Removed LNI 192.168.204.157@tcp [ 2086.192291] Key type .llcrypt unregistered [ 2086.193412] Key type ._llcrypt unregistered [ 2095.276071] Key type ._llcrypt registered [ 2095.277073] Key type .llcrypt registered [ 2095.309115] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2095.561232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.567827] alg: No test for adler32 (adler32-zlib) [ 2096.411822] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2096.491075] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2098.071134] Key type lgssc registered [ 2098.404133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2100.965583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2100.972397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.126763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.441314] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2106.289252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2106.415834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2107.734621] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2109.538773] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2110.705555] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2112.919900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2113.037846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.926595] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2116.135764] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2116.688022] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2127.397641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2134.454897] Lustre: *** cfs_fail_loc=159, val=0*** [ 2150.900608] Lustre: lustre-MDT0000: Client 4bc45642-e3a1-4ebd-a1d3-29e136a68919 (at 192.168.204.57@tcp) reconnecting [ 2152.928795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2152.929373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.936631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2152.938995] Lustre: Skipped 1 previous similar message [ 2158.048936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.050651] Lustre: Skipped 1 previous similar message [ 2158.188105] LustreError: 141813:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2158.241900] Lustre: server umount lustre-OST0000 complete [ 2162.655969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2162.658275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.661490] Lustre: Skipped 1 previous similar message [ 2162.663128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.560249] LustreError: 142014:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2165.562720] LustreError: 142014:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2165.588153] Lustre: server umount lustre-MDT0000 complete [ 2166.765159] LustreError: 139392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815080 with bad export cookie 5747834284385761125 [ 2166.768838] LustreError: 139392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2166.768848] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.828163] LustreError: 142217:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2166.830304] LustreError: 142217:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2166.909630] Lustre: server umount lustre-MDT0001 complete [ 2170.495411] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2171.519322] Key type lgssc unregistered [ 2171.650391] LNet: 142747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.653944] LNetError: 142747:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.663308] LNet: Removed LNI 192.168.204.157@tcp [ 2171.960726] Key type .llcrypt unregistered [ 2171.962036] Key type ._llcrypt unregistered [ 2180.724223] Key type ._llcrypt registered [ 2180.725672] Key type .llcrypt registered [ 2180.764963] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2181.074314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.081235] alg: No test for adler32 (adler32-zlib) [ 2181.926829] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2182.004660] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2183.583155] Key type lgssc registered [ 2183.964513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2186.557838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2186.565897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.722436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.032650] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2192.033286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2192.192524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2193.561127] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2195.458395] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2196.715489] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.008404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2199.135723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.182330] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2202.214218] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2202.902785] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2213.479455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2231.203878] Lustre: lustre-OST0000: evicting 3c7e55b2-88b3-4545-8c09-2a3e704a0b9e (at 192.168.204.57@tcp) by administrative request [ 2231.208587] Lustre: Skipped 1 previous similar message [ 2258.913180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2258.915747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.919726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2259.937487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2259.938341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2259.938835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.945628] Lustre: Skipped 1 previous similar message [ 2261.420198] LustreError: 146923:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2261.480964] Lustre: server umount lustre-OST0000 complete [ 2264.031791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.034196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.039701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.176401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.178559] Lustre: Skipped 3 previous similar messages [ 2275.296205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.298214] Lustre: Skipped 1 previous similar message [ 2277.343123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2277.378416] LustreError: 147124:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2277.380631] LustreError: 147124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2277.404756] Lustre: server umount lustre-MDT0000 complete [ 2278.793367] LustreError: 144098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815192 with bad export cookie 18425904460805533579 [ 2278.795611] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2278.799027] LustreError: 144098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2278.882808] LustreError: 147330:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2278.885286] LustreError: 147330:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2278.965592] Lustre: server umount lustre-MDT0001 complete [ 2282.736885] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2283.903588] Key type lgssc unregistered [ 2284.043408] LNet: 147861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.046422] LNetError: 147861:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.053324] LNet: Removed LNI 192.168.204.157@tcp [ 2284.374050] Key type .llcrypt unregistered [ 2284.375231] Key type ._llcrypt unregistered [ 2310.417110] Key type ._llcrypt registered [ 2310.418503] Key type .llcrypt registered [ 2310.461817] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2310.939465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.983196] alg: No test for adler32 (adler32-zlib) [ 2311.839323] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2311.945494] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2313.535157] Key type lgssc registered [ 2313.948969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2316.804955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2316.814559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.996319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.379507] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2322.413215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2322.555057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2324.053512] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2325.971475] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2327.263216] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2329.670623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2329.804865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2331.817772] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2331.882371] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2333.731869] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2343.142842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2347.487693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2347.490429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.495894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.514273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.514634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.515780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2349.230113] LustreError: 152577:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2349.295587] Lustre: server umount lustre-OST0000 complete [ 2352.703084] LustreError: 152778:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2352.705702] LustreError: 152778:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2352.733617] Lustre: server umount lustre-MDT0000 complete [ 2353.121061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2353.123550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.127707] LustreError: 150302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.133742] LustreError: 150302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2353.632828] LustreError: 151925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.041785] LustreError: 150280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815267 with bad export cookie 8925028631422420561 [ 2354.045089] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.045296] LustreError: 150280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2354.115529] LustreError: 152981:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2354.117769] LustreError: 152981:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2354.222664] Lustre: server umount lustre-MDT0001 complete [ 2357.903420] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2358.974700] Key type lgssc unregistered [ 2359.111471] LNet: 153512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.114647] LNetError: 153512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.123295] LNet: Removed LNI 192.168.204.157@tcp [ 2359.469919] Key type .llcrypt unregistered [ 2359.470961] Key type ._llcrypt unregistered [ 2369.223323] Key type ._llcrypt registered [ 2369.225144] Key type .llcrypt registered [ 2369.265109] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2369.670899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2369.767264] alg: No test for adler32 (adler32-zlib) [ 2370.624335] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2370.712941] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2372.295153] Key type lgssc registered [ 2372.667507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.132579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2377.667155] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 01:34:50 (1759815290) [ 2383.045271] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 2386.435136] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2389.818829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2392.178854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2394.237206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2396.555104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2398.932448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2398.954445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2398.959790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.042128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2400.052884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2400.090135] Lustre: lustre-MDT0000: new disk, initializing [ 2400.114528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.121236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2401.460077] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2405.703425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2405.728424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2405.762439] Lustre: 157705:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2405.792412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2405.794296] Lustre: Skipped 1 previous similar message [ 2405.834230] Lustre: lustre-MDT0001: new disk, initializing [ 2405.872077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2405.887065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2405.891402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2407.207770] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2410.312445] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2411.471375] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2413.742985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2413.764726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2413.866787] Lustre: lustre-OST0000: new disk, initializing [ 2413.868776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2413.899063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.653370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2414.657958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2414.697836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2416.003109] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2419.277866] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2424.556086] LustreError: 160288:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2424.619438] Lustre: server umount lustre-OST0000 complete [ 2426.335897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2426.338569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.343290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.920414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.920838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.040335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.042459] Lustre: Skipped 2 previous similar messages [ 2436.066225] LustreError: 160489:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2436.067947] LustreError: 160489:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2436.090794] Lustre: server umount lustre-MDT0000 complete [ 2437.381260] LustreError: 157696:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815350 with bad export cookie 9126232160970497662 [ 2437.385044] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.386850] LustreError: 157696:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2446.820355] LustreError: 160692:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2446.823984] LustreError: 160692:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2446.909989] Lustre: server umount lustre-MDT0001 complete [ 2450.750782] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2451.888986] Key type lgssc unregistered [ 2452.034367] LNet: 161224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.037118] LNetError: 161224:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.045273] LNet: Removed LNI 192.168.204.157@tcp [ 2452.381921] Key type .llcrypt unregistered [ 2452.383183] Key type ._llcrypt unregistered [ 2461.540530] Key type ._llcrypt registered [ 2461.541499] Key type .llcrypt registered [ 2461.579776] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2461.904326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.941097] alg: No test for adler32 (adler32-zlib) [ 2462.811619] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2462.904675] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2464.487179] Key type lgssc registered [ 2464.845717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2467.546288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2467.553859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.708331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2469.990050] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2472.977652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2473.100171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.366941] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2476.109995] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2477.280572] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2479.664053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2479.779799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.648024] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2483.489788] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2495.143239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2501.393728] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2501.402819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.410243] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2501.414451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2501.419809] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2501.759186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.091234] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2502.094136] Lustre: Skipped 1 previous similar message [ 2502.100325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.104141] Lustre: Skipped 1 previous similar message [ 2502.105582] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2502.107563] Lustre: Skipped 1 previous similar message [ 2502.109175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.112273] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2502.114854] Lustre: Skipped 1 previous similar message [ 2502.419227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.726936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.040903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.359212] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2503.363429] Lustre: Skipped 3 previous similar messages [ 2503.370821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.374180] Lustre: Skipped 3 previous similar messages [ 2503.375711] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2503.377624] Lustre: Skipped 3 previous similar messages [ 2503.379081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.382369] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2503.384019] Lustre: Skipped 3 previous similar messages [ 2503.682037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.987995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.307333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.637563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.958028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.286218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.591554] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2505.593289] Lustre: Skipped 6 previous similar messages [ 2505.598211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.601847] Lustre: Skipped 6 previous similar messages [ 2505.603362] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2505.605299] Lustre: Skipped 6 previous similar messages [ 2505.606711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.610309] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2505.612186] Lustre: Skipped 6 previous similar messages [ 2505.926255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.247775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.549244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.858819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.188753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.512574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.857623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.169248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.511812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.813357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.122564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.428387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.754993] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2509.757160] Lustre: Skipped 12 previous similar messages [ 2509.763412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.767881] Lustre: Skipped 12 previous similar messages [ 2509.770152] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2509.773797] Lustre: Skipped 12 previous similar messages [ 2509.775729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.779202] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2509.781502] Lustre: Skipped 12 previous similar messages [ 2510.087229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.392962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.707568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.054232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.374518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.687890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.015891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.334559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.662767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.986437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.311873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.643274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.945329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.254141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.557360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.883770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.192624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.491149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.797052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.105377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.417209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.728174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.032523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.362592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.697972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.014565] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2518.016614] Lustre: Skipped 25 previous similar messages [ 2518.021465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2518.025247] Lustre: Skipped 25 previous similar messages [ 2518.026682] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2518.029111] Lustre: Skipped 25 previous similar messages [ 2518.030851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.034200] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2518.036867] Lustre: Skipped 25 previous similar messages [ 2518.336765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.654409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.955867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.253504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.555498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.859563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.147302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.444348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.731424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.035982] LustreError: 162617:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2521.039756] LustreError: 162617:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2521.040213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.328701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.608731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.907073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.203806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.495795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.791333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.082929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.394183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.699890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.996160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.281317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.576765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.872373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.168795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.491465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.808884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.119664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.424629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.726389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.035316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.323684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.621871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.917886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.229027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.523184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.831163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.124453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.434875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.729347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.025893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.343431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.655379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.959889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.278671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.585410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.895393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.185039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.510798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.827403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.181786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.497180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.818631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.113754] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2534.116407] Lustre: Skipped 52 previous similar messages [ 2534.121568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.125010] Lustre: Skipped 52 previous similar messages [ 2534.126583] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2534.130102] Lustre: Skipped 52 previous similar messages [ 2534.131668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.135102] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2534.137024] Lustre: Skipped 52 previous similar messages [ 2534.468647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.773313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.069656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.360030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.682533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.983161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.281610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.574979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.882465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.191111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.505629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.811369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.125993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.416318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.715862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.033660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.367507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.683853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.990428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.308402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.631704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.935347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.253263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.583159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.925370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.252749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.570317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.875165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.196059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.518082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.859027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.197957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.525951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.831687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.159592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.489822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.816869] LustreError: 162617:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2545.819927] LustreError: 162617:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2545.821376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.123217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.428128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.735908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.081128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.402152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.737878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.048951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.385567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.708190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.041046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.368869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.709036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.020190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.329520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.649686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.969326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.305455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.630244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.954248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.277158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.583759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.897314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.192607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.499194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.801084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.111099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.406968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.737817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.049910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.367073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.692517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.021058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.317627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.619078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.931314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.234296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.544449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.850845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.148691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.468967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.784311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.091949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.406107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.740330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.086227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.391371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.698846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.031649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.323294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.616204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.917742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.236301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.546311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.869505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.183900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.511079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.820228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.149588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.474171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.809635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.129698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.443291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.762126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.096314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.419029] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2566.421390] Lustre: Skipped 101 previous similar messages [ 2566.427769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.432245] Lustre: Skipped 101 previous similar messages [ 2566.434037] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2566.435864] Lustre: Skipped 101 previous similar messages [ 2566.437656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.443611] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2566.447339] Lustre: Skipped 101 previous similar messages [ 2566.768116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.086747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.431672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.750203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.083976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.425675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.763598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.071945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.369374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.669389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.986269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.287797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.593590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.880071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.170296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.456556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.746700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.025783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.334666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.622437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.916253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.226239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.543585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.846165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.131461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.429558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.761435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.052298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.365683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.662004] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.972776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.268826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.569578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.882734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.170308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.446048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.724071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.007384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.306173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.603151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.899870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.174174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.461709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.745433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.039247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.320836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.614479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.890707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.161321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.460590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.760067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.062170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.369935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.662355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.960649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.236186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.517576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.796725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.074331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.352034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.629571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.901324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.208762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.499068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.786311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.072066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.367768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.632554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.903137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.168198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.456784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.757943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.069886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.363921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.675926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.945887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.234443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.524963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.831088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.116888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.405148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.716830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.064134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.357550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.636609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.904298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.192240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.465746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.743553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.009843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.281961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.552658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.828463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.136878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.460426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.779800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2595.083875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2595.390984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2595.721958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.039520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.347245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.664375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.007689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.320689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.638979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.949986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.290347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.625184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.965986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.285165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.608729] LustreError: 162617:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2599.613037] LustreError: 162617:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2599.614322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.936328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.273497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.590233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.930442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.251218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.572337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.879488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.199211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.527387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.845898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.190392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.582128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.914890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.239674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.549971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.882379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.193338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.514756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.832230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.170604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.493546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.830048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.178911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.519652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.846530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.163836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.488066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.832904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.145621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.454821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.801391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.144509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.458852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.767240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.084548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.410454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.726455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.034854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.338164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.676689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.988949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.312909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.627681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.956666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.268169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.569992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.875148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.183940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.497182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.797586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.100269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.426743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.734364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.037478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.328900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.641537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.968467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.299787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.612423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.933477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.246614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.586873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.901185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.250380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.589532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.930506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.254965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.591538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.900336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.205956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.507822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.836434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.167186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.480590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.789678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.104296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.407255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.714347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.028234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.339299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.639935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.949718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.266617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.572023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.860756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.157311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.461176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.773394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.093084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.422844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.739522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.101793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.425321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.741972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.061595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.403281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.713728] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2630.715857] Lustre: Skipped 207 previous similar messages [ 2630.720651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.724440] Lustre: Skipped 207 previous similar messages [ 2630.725931] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2630.727833] Lustre: Skipped 207 previous similar messages [ 2630.729129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.732952] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2630.735583] Lustre: Skipped 207 previous similar messages [ 2631.038159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.335628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.652292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.950770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.254396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.573771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.889429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.184828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.474412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.763609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.091920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.380679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.662145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.962406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.276605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.573227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.871695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.153550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.451186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.748026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.054276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.349105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.666466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.963173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2638.268127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2638.584451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2638.924391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.236095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.568298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.898353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.252022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.580094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.895250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.221961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.575260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.915703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.246937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.596232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.921884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.255539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.603360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.913835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.224364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.526599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.858099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.158848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.445351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.738253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.044214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.340662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.628915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.920830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2647.227572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2647.521872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2647.843522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.160496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.490683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.799419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.103853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.404095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.721523] LustreError: 162617:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2649.726225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.726789] LustreError: 162617:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2650.051460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2650.391473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2650.705484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2667.511765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2667.514180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2667.514343] LustreError: Skipped 4 previous similar messages [ 2669.536171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2673.625091] LustreError: 189749:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2673.722585] Lustre: server umount lustre-OST0000 complete [ 2678.239899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2678.243823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2678.246031] Lustre: Skipped 1 previous similar message [ 2681.092046] LustreError: 189950:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2681.095048] LustreError: 189950:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2681.149502] Lustre: server umount lustre-MDT0000 complete [ 2682.456199] LustreError: 189951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815595 with bad export cookie 8023703343118199042 [ 2682.459170] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2682.461348] LustreError: 189951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2682.531075] LustreError: 190153:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2682.533268] LustreError: 190153:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2682.617406] Lustre: server umount lustre-MDT0001 complete [ 2686.365696] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2687.407655] Key type lgssc unregistered [ 2687.542335] LNet: 190684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.545275] LNetError: 190684:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2687.557312] LNet: Removed LNI 192.168.204.157@tcp [ 2687.840987] Key type .llcrypt unregistered [ 2687.842038] Key type ._llcrypt unregistered [ 2696.650153] Key type ._llcrypt registered [ 2696.651191] Key type .llcrypt registered [ 2696.691313] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2697.136662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2697.145340] alg: No test for adler32 (adler32-zlib) [ 2698.018231] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2698.113560] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2699.703219] Key type lgssc registered [ 2700.102258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2702.753402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2702.761080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.926204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.252424] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2708.048451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2708.178107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2709.516509] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2711.278403] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2712.444052] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2714.696215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2714.806038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.649906] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2718.325767] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2730.152103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10739 to 0x280000401:10785) [ 2735.902797] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2736.530324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.534473] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2736.538152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.541191] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2736.541209] LustreError: 192077:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2738.655893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.658686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.664346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.183703] Lustre: lustre-OST0000: Not available for connect from 192.168.204.57@tcp (stopping) [ 2740.706809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.706962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.275249] LustreError: 194479:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2743.341084] Lustre: server umount lustre-OST0000 complete [ 2756.615052] LustreError: 192077:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2771.360689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2771.461350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.347095] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2775.151760] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2779.813511] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 2780.707405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.712882] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2780.991141] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93fe60675800.ost_server_uuid 50 [ 2781.488958] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93fe60675800.ost_server_uuid in FULL state after 0 sec [ 2785.761200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.766395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.768111] Lustre: Skipped 1 previous similar message [ 2788.586645] LustreError: 195796:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2788.588924] LustreError: 195796:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2788.650024] Lustre: server umount lustre-OST0000 complete [ 2790.368030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2790.370457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.375683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2802.144491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2802.147816] Lustre: Skipped 6 previous similar messages [ 2804.191113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2804.225264] LustreError: 195997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2804.227544] LustreError: 195997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2804.265278] Lustre: server umount lustre-MDT0000 complete [ 2805.581310] LustreError: 193676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815719 with bad export cookie 651265734398509953 [ 2805.585034] LustreError: 193676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2805.585306] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2805.734303] Lustre: server umount lustre-MDT0001 complete [ 2809.543543] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2810.647377] Key type lgssc unregistered [ 2810.787410] LNet: 196730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2810.790082] LNetError: 196730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2810.796305] LNet: Removed LNI 192.168.204.157@tcp [ 2811.129774] Key type .llcrypt unregistered [ 2811.131567] Key type ._llcrypt unregistered [ 2822.920578] Key type ._llcrypt registered [ 2822.921686] Key type .llcrypt registered [ 2822.961747] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2823.258289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2823.272965] alg: No test for adler32 (adler32-zlib) [ 2824.132573] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2824.233254] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2825.815226] Key type lgssc registered [ 2826.202714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2829.088201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2829.096064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.238104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.543224] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2834.341329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2834.467790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.828903] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2836.815155] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2839.024282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2839.138333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.169268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10787 to 0x280000401:10817) [ 2841.728160] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2844.652298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2844.681295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2844.735269] Lustre: lustre-OST0001: new disk, initializing [ 2844.737107] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2844.765652] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.619770] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2851.840502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2851.844933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2851.877667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2852.038725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2857.711056] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2872.294177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.296330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2872.297973] Lustre: Skipped 3 previous similar messages [ 2872.301211] Lustre: Skipped 2 previous similar messages [ 2877.123951] LustreError: 201780:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2877.161383] Lustre: server umount lustre-MDT0000 complete [ 2877.409315] LustreError: 198417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.414860] LustreError: 198417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2878.449097] LustreError: 201439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815791 with bad export cookie 12021891764887502535 [ 2878.452481] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2878.453626] LustreError: 201439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2878.521122] LustreError: 201982:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2878.523373] LustreError: 201982:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2878.601914] Lustre: server umount lustre-MDT0001 complete [ 2889.715099] LustreError: 202182:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2889.716955] LustreError: 202182:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2889.731130] Lustre: server umount lustre-OST0000 complete [ 2901.232897] LustreError: 202385:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2901.235562] LustreError: 202385:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2901.302910] Lustre: server umount lustre-OST0001 complete [ 2902.349223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2902.374745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2902.724842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2902.754644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2903.120780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2903.147252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2903.502511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2903.535574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2908.273732] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2911.866847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.021149] LustreError: 203793:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.042103] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.336642] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2916.245343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2917.838421] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2918.827738] Lustre: 204901:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2921.207413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2921.364751] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.367828] Lustre: Skipped 1 previous similar message [ 2922.403489] LustreError: 205429:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.412299] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10819 to 0x280000401:10849) [ 2923.507043] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2926.503115] LustreError: 205429:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.513570] LustreError: 205429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2926.604934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2927.799856] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2928.725962] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2931.846919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2933.081345] Lustre: 206750:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2947.040832] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2947.043611] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.048743] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.050380] Lustre: Skipped 1 previous similar message [ 2948.605950] LustreError: 207058:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2948.607989] LustreError: 207058:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2948.638594] Lustre: server umount mylustre-MDT0000 complete [ 2949.870140] LustreError: 203770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815863 with bad export cookie 12021891764887505328 [ 2949.873610] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2949.874730] LustreError: 203770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2950.012196] Lustre: server umount mylustre-MDT0001 complete [ 2960.882297] LustreError: 207460:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 2960.884184] LustreError: 207460:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 2960.898788] Lustre: server umount mylustre-OST0000 complete [ 2972.406532] Lustre: server umount mylustre-OST0001 complete [ 2973.445511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2973.476145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2973.842877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2973.872569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2974.257305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2974.287282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2974.653634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2974.686108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2979.293355] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2982.719210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.879951] LustreError: 209069:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2982.903654] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.905388] Lustre: Skipped 1 previous similar message [ 2984.280411] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2987.239585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2988.781387] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2989.771667] Lustre: 210179:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2991.996943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2993.124509] LustreError: 210537:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.130517] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10819 to 0x280000401:10881) [ 2994.022849] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2996.879557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2997.998479] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 2998.937445] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3002.179791] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3003.413118] Lustre: 212026:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3013.087811] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3013.090506] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.093792] Lustre: Skipped 3 previous similar messages [ 3013.095302] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.098026] Lustre: Skipped 3 previous similar messages [ 3016.513692] LustreError: 212275:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3016.517281] LustreError: 212275:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3016.546677] Lustre: server umount tfs-MDT0000 complete [ 3017.785282] LustreError: 209045:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815931 with bad export cookie 12021891764887507218 [ 3017.788752] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3017.788761] LustreError: 209045:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3017.932750] Lustre: server umount tfs-MDT0001 complete [ 3029.443529] Lustre: server umount tfs-OST0000 complete [ 3041.076836] Lustre: server umount tfs-OST0001 complete [ 3042.158684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3042.188571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3042.549947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3042.582161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3042.969438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3043.003606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3043.382838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3043.411527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3048.125427] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3051.694327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.869515] LustreError: 214286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.892923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.895360] Lustre: Skipped 3 previous similar messages [ 3053.171588] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3056.133933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3057.754806] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3058.745136] Lustre: 215395:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3061.043966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3062.184584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10819 to 0x280000401:10913) [ 3063.165562] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3066.197268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3067.319845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3068.368658] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3071.539531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3072.761565] Lustre: 217242:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3080.201570] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 01:46:32 (1759815992) [ 3091.939423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.943770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.944853] Lustre: Skipped 5 previous similar messages [ 3091.947192] Lustre: Skipped 4 previous similar messages [ 3097.059241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.061025] Lustre: Skipped 4 previous similar messages [ 3106.271134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3106.307658] LustreError: 217613:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3106.310583] LustreError: 217613:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3106.342733] Lustre: server umount lustre-MDT0000 complete [ 3107.298377] LustreError: 214286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.306402] LustreError: 214286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3107.586734] LustreError: 215014:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816021 with bad export cookie 12021891764887508884 [ 3107.589743] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3107.592312] LustreError: 215014:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3112.416783] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.417695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3112.420394] Lustre: Skipped 2 previous similar messages [ 3112.424926] Lustre: Skipped 5 previous similar messages [ 3113.823483] Lustre: server umount lustre-MDT0001 complete [ 3121.212912] Lustre: server umount lustre-OST0000 complete [ 3128.554277] Lustre: server umount lustre-OST0001 complete [ 3130.708556] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 3134.062347] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3137.798867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.040174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3142.089887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3144.187062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3146.478051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3146.500117] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3146.500117] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3146.500117] [ 3146.509388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3146.589989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3146.601529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3146.639971] Lustre: lustre-MDT0000: new disk, initializing [ 3146.666265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.668652] Lustre: Skipped 3 previous similar messages [ 3146.674713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3147.998958] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3152.301060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3152.327626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3152.363221] Lustre: 220398:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3152.386601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3152.388456] Lustre: Skipped 1 previous similar message [ 3152.426046] Lustre: lustre-MDT0001: new disk, initializing [ 3152.472839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3152.476866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3153.819513] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3158.117861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3158.144725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3158.244626] Lustre: lustre-OST0000: new disk, initializing [ 3158.247127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3160.059887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3160.064427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3160.102080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3160.359903] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3164.696913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3164.722868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3164.782652] Lustre: lustre-OST0001: new disk, initializing [ 3164.785192] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3166.797874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3166.801129] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3166.841231] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3166.881052] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3171.397521] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 01:48:03 (1759816083) [ 3175.048145] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 01:48:07 (1759816087) [ 3187.683586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.685425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3187.688730] Lustre: Skipped 1 previous similar message [ 3192.493882] LustreError: 223822:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3192.496919] LustreError: 223822:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3192.517734] Lustre: server umount lustre-OST0000 complete [ 3192.805425] LustreError: 221936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.811269] LustreError: 221936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3208.159155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3208.239969] Lustre: server umount lustre-MDT0000 complete [ 3209.563336] LustreError: 223723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816123 with bad export cookie 12021891764887510438 [ 3209.567231] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3209.568669] LustreError: 223723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3213.280420] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.286185] Lustre: Skipped 3 previous similar messages [ 3215.775074] Lustre: server umount lustre-MDT0001 complete [ 3219.830124] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3229.608303] Lustre: server umount lustre-OST0001 complete [ 3231.818767] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 3238.278907] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3242.051792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3244.247785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3246.359209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3248.581991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3250.798148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3250.820839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.905488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3250.918226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3250.963458] Lustre: lustre-MDT0000: new disk, initializing [ 3250.992049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3252.322409] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3256.477786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3256.503434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3256.542345] Lustre: 227284:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3256.651822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3257.954829] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3261.003029] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3262.274985] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3264.563079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3264.587321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3265.913343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3265.953744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3266.569234] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3269.652637] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3276.256906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3276.259173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.260271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.263932] Lustre: Skipped 1 previous similar message [ 3276.265660] Lustre: Skipped 12 previous similar messages [ 3278.076409] Lustre: server umount lustre-OST0000 complete [ 3285.533158] Lustre: server umount lustre-MDT0000 complete [ 3286.845263] LustreError: 227275:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816200 with bad export cookie 12021891764887511908 [ 3286.848960] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.851055] LustreError: 227275:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3287.082556] Lustre: server umount lustre-MDT0001 complete [ 3291.200031] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3292.333649] Key type lgssc unregistered [ 3292.493464] LNet: 230660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3292.497416] LNetError: 230660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3292.509449] LNet: Removed LNI 192.168.204.157@tcp [ 3292.849099] Key type .llcrypt unregistered [ 3292.850799] Key type ._llcrypt unregistered [ 3302.595558] Key type ._llcrypt registered [ 3302.597124] Key type .llcrypt registered [ 3302.639345] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3302.962088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3303.013244] alg: No test for adler32 (adler32-zlib) [ 3303.868898] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3303.963369] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3305.551148] Key type lgssc registered [ 3305.914502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3308.707649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3308.717175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3309.886780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.302463] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3314.250511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3314.378265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3315.854683] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3317.702429] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3319.014820] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3321.407309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3321.529909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.508814] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3325.411768] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3325.749909] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3328.992650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3328.996511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3329.003071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3332.458100] LustreError: 234378:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3332.516702] Lustre: server umount lustre-OST0000 complete [ 3334.112831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.113754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.117249] Lustre: Skipped 1 previous similar message [ 3334.120174] Lustre: Skipped 1 previous similar message [ 3339.231984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3339.234124] Lustre: Skipped 1 previous similar message [ 3339.965891] LustreError: 234587:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3339.969674] LustreError: 234587:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3339.996813] Lustre: server umount lustre-MDT0000 complete [ 3341.310207] LustreError: 232061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816254 with bad export cookie 9322140258398751276 [ 3341.313597] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.314291] LustreError: 232061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3341.397644] LustreError: 234788:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3341.399661] LustreError: 234788:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3341.482092] Lustre: server umount lustre-MDT0001 complete [ 3345.186437] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3346.299386] Key type lgssc unregistered [ 3346.437382] LNet: 235320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3346.441470] LNetError: 235320:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3346.449290] LNet: Removed LNI 192.168.204.157@tcp [ 3346.766984] Key type .llcrypt unregistered [ 3346.768250] Key type ._llcrypt unregistered [ 3353.465711] Key type ._llcrypt registered [ 3353.467249] Key type .llcrypt registered [ 3353.506849] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3353.831210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3353.859121] alg: No test for adler32 (adler32-zlib) [ 3354.718523] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3354.815311] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3356.399181] Key type lgssc registered [ 3356.775910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3359.932063] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3359.941520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3375.391282] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3380.511221] LustreError: 236470:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 3407.135183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.168028] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3411.057489] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3411.809299] LustreError: 237133:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3411.868622] Lustre: server umount lustre-OST0000 complete [ 3417.673756] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3418.792145] Key type lgssc unregistered [ 3418.938413] LNet: 237968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3418.942304] LNetError: 237968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3418.952317] LNet: Removed LNI 192.168.204.157@tcp [ 3419.266122] Key type .llcrypt unregistered [ 3419.267757] Key type ._llcrypt unregistered [ 3433.502108] Key type ._llcrypt registered [ 3433.503774] Key type .llcrypt registered [ 3433.543335] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3433.911881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3433.948571] alg: No test for adler32 (adler32-zlib) [ 3434.814704] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3434.904523] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3436.487167] Key type lgssc registered [ 3436.876070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3441.436138] loop0: detected capacity change from 0 to 204800000 [ 3441.538919] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.544692] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.684674] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.690712] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.718762] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.744047] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.763955] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.796656] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.828472] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3441.863428] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3442.230391] loop0: detected capacity change from 0 to 204800000 [ 3442.296234] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3442.573958] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3442.933234] loop0: detected capacity change from 0 to 204800000 [ 3443.092138] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3443.439330] loop0: detected capacity change from 0 to 204800000 [ 3443.603298] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3443.951124] loop0: detected capacity change from 0 to 204800000 [ 3443.963112] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3445.341510] loop0: detected capacity change from 0 to 204800000 [ 3445.353732] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3446.700796] loop0: detected capacity change from 0 to 204800000 [ 3446.710922] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3448.130993] loop0: detected capacity change from 0 to 204800000 [ 3448.142236] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3449.498513] loop0: detected capacity change from 0 to 204800000 [ 3449.534951] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3449.554789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3449.563578] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.591904] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3449.595654] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3449.599058] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3449.603913] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3449.608947] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3449.612843] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3449.617167] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3449.621214] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3449.625415] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3449.629619] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3449.633203] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3449.636867] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3449.640941] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3449.644972] Lustre: 241342:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3452.272041] LustreError: 241609:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3452.316734] Lustre: server umount MGS complete [ 3452.619629] loop0: detected capacity change from 0 to 204800000 [ 3452.665070] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.826412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3452.841152] Lustre: lustre-MDT0000: Aborting client recovery [ 3453.152735] loop1: detected capacity change from 0 to 204800000 [ 3453.216775] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3453.239075] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3453.260739] Lustre: 241778:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3453.265408] Lustre: 241778:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3453.270467] Lustre: 241778:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3453.274804] Lustre: 241778:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3453.279173] Lustre: 241778:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3453.283655] Lustre: 241778:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3453.287814] Lustre: 241778:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3453.295213] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3453.368912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.396039] Lustre: lustre-MDT0001: Aborting client recovery [ 3453.723121] loop2: detected capacity change from 0 to 204800000 [ 3453.783093] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3453.804638] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3453.820987] Lustre: 241862:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3453.824349] Lustre: 241862:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3453.827513] Lustre: 241862:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3453.830673] Lustre: 241862:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3453.833546] Lustre: 241862:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3453.837258] Lustre: 241862:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3453.840642] Lustre: 241862:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3453.888463] Lustre: Found index 0 for lustre-OST0000, updating log [ 3453.923728] Lustre: lustre-OST0000: Aborting recovery [ 3454.236812] loop3: detected capacity change from 0 to 204800000 [ 3454.277314] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3454.321611] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3454.338816] Lustre: 241949:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3454.342782] Lustre: 241949:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3454.349318] Lustre: 241949:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3454.355724] Lustre: 241949:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3454.360521] Lustre: 241949:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3454.363992] Lustre: 241949:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3454.370359] Lustre: 241949:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3454.411442] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3454.413565] Lustre: Skipped 1 previous similar message [ 3459.562410] LustreError: 241721:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 3462.626494] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3462.626796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3462.628746] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3462.633610] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3462.639838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3462.647951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3465.174238] LustreError: 241688:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 245us: rc = -2 [ 3477.985588] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.986095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3477.989587] Lustre: Skipped 3 previous similar messages [ 3477.991778] Lustre: Skipped 2 previous similar messages [ 3482.425652] LustreError: 242304:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3482.428123] LustreError: 242304:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3482.459946] Lustre: server umount lustre-MDT0000 complete [ 3482.769630] LustreError: 242305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816396 with bad export cookie 5934182277261114421 [ 3482.771270] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3482.772864] LustreError: 242305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3483.103573] LustreError: 241684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.103881] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.104309] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3483.110312] LustreError: 241684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3483.117434] Lustre: Skipped 1 previous similar message [ 3487.135564] LustreError: 241684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3487.135661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3487.141863] LustreError: 241684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3487.146172] Lustre: Skipped 2 previous similar messages [ 3489.018391] LustreError: 242357:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3489.021564] LustreError: 242357:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3489.103975] Lustre: server umount lustre-MDT0001 complete [ 3495.468066] LustreError: 242409:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3495.470685] LustreError: 242409:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3495.494336] Lustre: server umount lustre-OST0000 complete [ 3510.239085] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3510.254248] LustreError: 242460:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3510.256185] LustreError: 242460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3510.329664] Lustre: server umount lustre-OST0001 complete [ 3512.873652] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 01:53:45 (1759816425) [ 3522.027561] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 3525.177306] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3528.851755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.046660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3533.011614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3535.132311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3537.379545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3537.403450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.483104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3537.494091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3537.529346] Lustre: lustre-MDT0000: new disk, initializing [ 3537.554934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.561341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3538.803705] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3542.982077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3543.002485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3543.037852] Lustre: 246008:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3543.060246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3543.062346] Lustre: Skipped 1 previous similar message [ 3543.096537] Lustre: lustre-MDT0001: new disk, initializing [ 3543.128752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3543.141169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3543.144265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3544.426423] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3547.574995] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3548.821375] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3551.115395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.133781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3551.231389] Lustre: lustre-OST0000: new disk, initializing [ 3551.233508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3551.263501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3553.153511] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3556.286218] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3556.350040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3556.353694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3556.390979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3558.660509] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3564.978363] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3573.402234] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3582.050896] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 3622.880499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3622.882964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3622.883869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3622.886444] Lustre: Skipped 1 previous similar message [ 3627.245210] LustreError: 249713:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3627.247739] LustreError: 249713:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3627.314452] Lustre: server umount lustre-OST0000 complete [ 3630.047785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3630.050257] LustreError: Skipped 1 previous similar message [ 3630.051691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.056198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.058526] Lustre: Skipped 1 previous similar message [ 3634.650559] Lustre: server umount lustre-MDT0000 complete [ 3635.904220] LustreError: 247752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816549 with bad export cookie 5934182277261117123 [ 3635.907878] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3635.908242] LustreError: 247752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3636.054194] Lustre: server umount lustre-MDT0001 complete [ 3638.968977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.393835] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3642.108628] Lustre: server umount MGS complete [ 3645.954321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.112866] LustreError: 251682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.119951] LustreError: 251694:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3646.143269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3647.446595] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3650.254263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3651.724636] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3653.576928] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3654.792864] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3657.125757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3659.136794] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3660.956944] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3701.215975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.218563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.222158] Lustre: Skipped 1 previous similar message [ 3701.224066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.226106] Lustre: Skipped 2 previous similar messages [ 3704.493122] LustreError: 254359:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3704.495598] LustreError: 254359:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 3704.557338] Lustre: server umount lustre-OST0000 complete [ 3706.847840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3706.850336] LustreError: Skipped 1 previous similar message [ 3717.600591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.602541] Lustre: Skipped 8 previous similar messages [ 3720.159112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3720.227512] Lustre: server umount lustre-MDT0000 complete [ 3721.472590] LustreError: 254564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816634 with bad export cookie 5934182277261118936 [ 3721.475476] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.475623] LustreError: 254564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3721.622284] Lustre: server umount lustre-MDT0001 complete [ 3725.202516] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3726.303191] Key type lgssc unregistered [ 3726.450406] LNet: 255297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3726.453273] LNetError: 255297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3726.463357] LNet: Removed LNI 192.168.204.157@tcp [ 3726.768202] Key type .llcrypt unregistered [ 3726.769485] Key type ._llcrypt unregistered [ 3738.925591] Key type ._llcrypt registered [ 3738.926631] Key type .llcrypt registered [ 3738.967575] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 3745.205952] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3745.545265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3745.589251] alg: No test for adler32 (adler32-zlib) [ 3746.439461] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3746.522714] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3748.103173] Key type lgssc registered [ 3748.468791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3751.480623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3753.749761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3755.774846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3757.899990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3760.189533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3760.207754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3760.213279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.294943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3761.307804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3761.345830] Lustre: lustre-MDT0000: new disk, initializing [ 3761.370988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.379162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3762.711290] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3766.874686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3766.899753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3766.932304] Lustre: 259200:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3766.954293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3766.956229] Lustre: Skipped 1 previous similar message [ 3766.992379] Lustre: lustre-MDT0001: new disk, initializing [ 3767.029160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3767.041949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3767.047315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3768.383085] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3771.484158] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3772.717219] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3774.993652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3775.019734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3775.120047] Lustre: lustre-OST0000: new disk, initializing [ 3775.123217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3775.156550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.439432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3776.442274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3776.482326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3777.103147] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3780.277181] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3783.722638] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3794.009466] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3802.382882] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3811.977074] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 3848.161365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3848.163125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.165022] LustreError: Skipped 1 previous similar message [ 3848.165812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3848.169419] Lustre: Skipped 1 previous similar message [ 3853.281966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.283668] Lustre: Skipped 2 previous similar messages [ 3853.868113] LustreError: 263059:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3853.927299] Lustre: server umount lustre-OST0000 complete [ 3858.400658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.404735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3858.406655] Lustre: Skipped 1 previous similar message [ 3861.310507] LustreError: 263261:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3861.313567] LustreError: 263261:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3861.344950] Lustre: server umount lustre-MDT0000 complete [ 3862.559247] LustreError: 262895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816776 with bad export cookie 12893283440774913655 [ 3862.563758] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3862.618852] LustreError: 263462:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3862.622197] LustreError: 263462:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3862.703127] Lustre: server umount lustre-MDT0001 complete [ 3865.480164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.784215] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3868.438969] LustreError: 264403:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3868.441784] LustreError: 264403:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3868.465953] Lustre: server umount MGS complete [ 3872.400762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3872.560190] LustreError: 265037:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3872.581599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.837265] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3876.528851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3876.647941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3877.919378] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3879.719934] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3880.843390] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3883.019266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3884.955566] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3886.714132] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3919.333333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.333443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3919.334586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3919.336927] Lustre: Skipped 2 previous similar messages [ 3924.449444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3924.451458] Lustre: Skipped 1 previous similar message [ 3925.290114] LustreError: 267705:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3925.292456] LustreError: 267705:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3925.352882] Lustre: server umount lustre-OST0000 complete [ 3928.031890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3928.034035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.762826] Lustre: server umount lustre-MDT0000 complete [ 3934.003162] LustreError: 265005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816847 with bad export cookie 12893283440774915580 [ 3934.006454] LustreError: 265005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3934.007118] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3934.078107] LustreError: 268107:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3934.080467] LustreError: 268107:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3934.160351] Lustre: server umount lustre-MDT0001 complete [ 3938.050737] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3939.073983] Key type lgssc unregistered [ 3939.209364] LNet: 268638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3939.212195] LNetError: 268638:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3939.221314] LNet: Removed LNI 192.168.204.157@tcp [ 3939.532114] Key type .llcrypt unregistered [ 3939.533082] Key type ._llcrypt unregistered [ 3946.136933] Key type ._llcrypt registered [ 3946.137953] Key type .llcrypt registered [ 3946.172315] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 3952.270220] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3952.682472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3952.715078] alg: No test for adler32 (adler32-zlib) [ 3953.560713] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3953.640691] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3955.223165] Key type lgssc registered [ 3955.593194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3958.592836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.603146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3963.787940] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3966.644373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3968.927637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3968.950492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3968.956162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.050558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3970.062460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3970.105889] Lustre: lustre-MDT0000: new disk, initializing [ 3970.132569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.139585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3971.445152] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3975.501047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3975.524566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3975.613815] Lustre: lustre-OST0000: new disk, initializing [ 3975.616352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3975.618069] Lustre: Skipped 1 previous similar message [ 3975.639729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.427850] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3977.983488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3977.988461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3977.998970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4147.168385] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.172416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4150.014663] LustreError: 273558:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4150.163911] Lustre: server umount lustre-MDT0000 complete [ 4151.406938] LustreError: 271812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817064 with bad export cookie 13895319306520630438 [ 4151.408674] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4151.410132] LustreError: 271812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4151.425813] LustreError: 273761:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4151.427506] LustreError: 273761:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4151.493316] Lustre: server umount lustre-OST0000 complete [ 4159.595468] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 4163.086431] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4166.606695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4168.980059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4171.037115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4173.234314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4175.506638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4175.530163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.605875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4175.616646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4175.652467] Lustre: lustre-MDT0000: new disk, initializing [ 4175.676060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.682450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4176.977134] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4181.443405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4181.465753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4181.495741] Lustre: 276798:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4181.519190] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4181.520971] Lustre: Skipped 1 previous similar message [ 4181.555818] Lustre: lustre-MDT0001: new disk, initializing [ 4181.584774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4181.595393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4181.598424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4182.880128] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4185.940899] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4187.142174] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4189.412794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4189.437955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4189.530816] Lustre: lustre-OST0000: new disk, initializing [ 4189.532854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4189.562831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.391392] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4194.494519] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4195.927257] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4199.928643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4199.931471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4199.957815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4200.094068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4201.365440] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4201.441628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4205.026167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4205.026931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.030035] LustreError: Skipped 1 previous similar message [ 4205.037412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4205.039755] Lustre: Skipped 1 previous similar message [ 4208.174122] LustreError: 279768:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4208.175893] LustreError: 279768:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4208.250860] Lustre: server umount lustre-OST0000 complete [ 4210.144748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.145299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.148425] Lustre: Skipped 2 previous similar messages [ 4210.153279] Lustre: Skipped 1 previous similar message [ 4215.264680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.266362] Lustre: Skipped 1 previous similar message [ 4215.617408] LustreError: 279969:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4215.619841] LustreError: 279969:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4215.660662] Lustre: server umount lustre-MDT0000 complete [ 4216.941899] LustreError: 276790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817130 with bad export cookie 13895319306523571215 [ 4216.947747] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4217.105400] Lustre: server umount lustre-MDT0001 complete [ 4218.299206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 4218.884482] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 02:05:31 (1759817131) [ 4221.191599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.363766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.685562] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4225.438865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4226.858775] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4228.648473] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4229.897674] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4232.205789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4234.135929] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4235.931470] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4238.164502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4238.187733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4238.238881] Lustre: lustre-OST0001: new disk, initializing [ 4238.240559] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4238.268883] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4238.271051] Lustre: Skipped 2 previous similar messages [ 4238.975953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4238.978932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4238.998741] LustreError: 283377:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4240.035328] LustreError: 283384:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4240.038464] LustreError: 283384:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 4240.108962] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4241.058495] LustreError: 283384:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4241.061474] LustreError: 283384:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4243.106766] LustreError: 283384:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4243.110973] LustreError: 283384:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 4243.784281] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4247.211494] LustreError: 283384:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4247.217394] LustreError: 283384:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 4247.946688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4249.072266] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93fe435a2800.ost_server_uuid 50 [ 4249.575565] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93fe435a2800.ost_server_uuid in FULL state after 0 sec [ 4250.815177] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93fe435a2800.ost_server_uuid 50 [ 4251.288540] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93fe435a2800.ost_server_uuid in FULL state after 0 sec [ 4252.327917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4262.883415] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4262.884316] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4262.886265] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4262.891265] Lustre: Skipped 1 previous similar message [ 4262.894316] Lustre: Skipped 1 previous similar message [ 4268.718144] LustreError: 284557:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4268.720118] LustreError: 284557:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4268.741508] Lustre: server umount lustre-OST0001 complete [ 4271.074156] LustreError: 282338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.075066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.076267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4271.076271] Lustre: Skipped 4 previous similar messages [ 4271.078442] LustreError: 282338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4271.081666] Lustre: Skipped 1 previous similar message [ 4276.193826] LustreError: 282339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.200034] LustreError: 282339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4276.514595] Lustre: server umount lustre-OST0000 complete [ 4281.312734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.316311] Lustre: Skipped 1 previous similar message [ 4291.553819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4291.556398] Lustre: Skipped 8 previous similar messages [ 4292.063108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4292.099621] LustreError: 284963:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4292.101344] LustreError: 284963:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4292.129283] Lustre: server umount lustre-MDT0000 complete [ 4293.357493] LustreError: 282361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817206 with bad export cookie 13895319306523572083 [ 4293.360364] LustreError: 282361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4293.362781] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4293.529851] Lustre: server umount lustre-MDT0001 complete [ 4297.376711] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4298.462428] Key type lgssc unregistered [ 4298.612526] LNet: 285698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4298.616207] LNetError: 285698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4298.623405] LNet: Removed LNI 192.168.204.157@tcp [ 4298.965066] Key type .llcrypt unregistered [ 4298.966120] Key type ._llcrypt unregistered [ 4308.283900] Key type ._llcrypt registered [ 4308.285017] Key type .llcrypt registered [ 4308.327175] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4308.770275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4308.815827] alg: No test for adler32 (adler32-zlib) [ 4309.690261] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4309.786937] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4311.375162] Key type lgssc registered [ 4311.730760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4314.440216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4314.446843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.649426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.048375] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4318.859297] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4321.115979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4321.267677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.621034] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4325.778823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4325.931456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.950294] LustreError: 288627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4326.953785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4326.955277] LustreError: 288627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4327.813788] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4329.612141] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4331.044930] LustreError: 288808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.050321] LustreError: 288808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4331.898289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4331.964458] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4332.135896] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4333.813754] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4335.616722] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4339.199599] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4339.302608] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4419.044958] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.049664] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4419.052108] Lustre: Skipped 1 previous similar message [ 4424.162731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4424.165728] Lustre: Skipped 1 previous similar message [ 4429.279157] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4429.283898] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4429.285446] Lustre: Skipped 1 previous similar message [ 4429.303804] LustreError: 290496:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4429.317080] Lustre: server umount lustre-OST0001 complete [ 4433.376609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4433.380257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.384427] Lustre: Skipped 1 previous similar message [ 4433.386235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4434.401613] LustreError: 288808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.405283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.405437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4434.406089] LustreError: 288808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4437.037288] LustreError: 290699:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4437.039057] LustreError: 290699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4437.107115] Lustre: server umount lustre-OST0000 complete [ 4439.007915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4439.010205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.014760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4439.016545] Lustre: Skipped 2 previous similar messages [ 4444.480576] LustreError: 290900:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4444.483082] LustreError: 290900:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4444.520903] Lustre: server umount lustre-MDT0000 complete [ 4444.640155] LustreError: 287076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.643864] LustreError: 287076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4445.805073] LustreError: 287055:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817359 with bad export cookie 2649142913297792590 [ 4445.810724] LustreError: 287055:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4445.811464] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4445.999941] Lustre: server umount lustre-MDT0001 complete [ 4449.727803] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4450.852574] Key type lgssc unregistered [ 4451.002677] LNet: 291632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4451.005072] LNetError: 291632:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4451.016444] LNet: Removed LNI 192.168.204.157@tcp [ 4451.340513] Key type .llcrypt unregistered [ 4451.341469] Key type ._llcrypt unregistered [ 4469.238604] Key type ._llcrypt registered [ 4469.239570] Key type .llcrypt registered [ 4469.277277] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4469.627612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4469.677116] alg: No test for adler32 (adler32-zlib) [ 4470.530551] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4470.618341] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4472.199205] Key type lgssc registered [ 4472.606288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4475.746193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4475.753916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.922299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4478.247629] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4481.044768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4481.160386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4482.450859] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4483.475591] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4485.715948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4485.882396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4486.885545] LustreError: 295339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.890908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4486.891049] LustreError: 295339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4487.930242] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4490.865217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4490.964346] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4492.027273] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 4492.985087] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4497.012946] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4500.629270] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4512.223982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4512.226022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.230033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.739559] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.741414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.743076] Lustre: Skipped 2 previous similar messages [ 4512.744968] Lustre: Skipped 1 previous similar message [ 4515.390212] LustreError: 297291:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4515.422869] Lustre: server umount lustre-MDT0000 complete [ 4516.594219] LustreError: 293862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817430 with bad export cookie 14465143331653844006 [ 4516.595284] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4516.598031] LustreError: 293862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4516.666699] LustreError: 297492:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4516.668461] LustreError: 297492:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4516.754969] Lustre: server umount lustre-MDT0001 complete [ 4528.179068] LustreError: 297693:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4528.180951] LustreError: 297693:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4528.195833] Lustre: server umount lustre-OST0000 complete [ 4539.573094] LustreError: 297894:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4539.576275] LustreError: 297894:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4539.668268] Lustre: server umount lustre-OST0001 complete [ 4545.321149] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4549.001426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.153233] LustreError: 299071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.172897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.469313] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4553.398226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4554.929339] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4555.959678] Lustre: 300178:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4558.363365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4558.505643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.507900] Lustre: Skipped 1 previous similar message [ 4559.525380] LustreError: 300534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.534850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4560.509206] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4562.598599] LustreError: 300714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.604878] LustreError: 300714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4563.485818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4565.564391] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4565.622927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 4568.732926] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4570.011821] Lustre: 302027:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4579.298312] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.300859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4579.301830] Lustre: Skipped 3 previous similar messages [ 4579.303558] Lustre: Skipped 3 previous similar messages [ 4579.306880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.150167] LustreError: 302132:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4582.151923] LustreError: 302132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4582.179896] Lustre: server umount lustre-MDT0000 complete [ 4583.442589] LustreError: 302030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817496 with bad export cookie 14465143331653845686 [ 4583.446275] LustreError: 302030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4583.448789] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.603967] Lustre: server umount lustre-MDT0001 complete [ 4595.184087] LustreError: 302535:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4595.186625] LustreError: 302535:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4595.201877] Lustre: server umount lustre-OST0000 complete [ 4605.697887] Lustre: server umount lustre-OST0001 complete [ 4606.908021] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 4607.408649] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 4607.934440] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 02:12:00 (1759817520) [ 4614.986241] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4618.066758] loop0: detected capacity change from 0 to 1099511627776 [ 4618.236324] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 4618.239112] *** VALIDATE xfs *** [ 4618.244634] XFS (loop0): Mounting V5 Filesystem [ 4618.255290] XFS (loop0): Ending clean mount [ 4618.666791] loop1: detected capacity change from 0 to 18691697672192 [ 4633.550453] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.271649] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 4641.972392] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4646.649766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4649.865615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4652.570266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4655.768298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.446038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.493229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.622124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4658.640371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4658.696637] Lustre: lustre-MDT0000: new disk, initializing [ 4658.729795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.733488] Lustre: Skipped 1 previous similar message [ 4658.741934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4660.380329] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4665.920959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4665.963256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4666.017385] Lustre: 307104:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4666.061947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4666.066777] Lustre: Skipped 1 previous similar message [ 4666.115713] Lustre: lustre-MDT0001: new disk, initializing [ 4666.183948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4666.192749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4667.839097] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4671.577981] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4673.430611] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4676.657221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4676.681901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4676.807139] Lustre: lustre-OST0000: new disk, initializing [ 4676.809603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4678.025403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4678.029824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4678.080066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4678.973766] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4683.181210] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4684.900729] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4684.980622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4686.716494] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4686.819453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4688.356824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4688.358796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.361138] LustreError: Skipped 1 previous similar message [ 4688.369688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4688.374269] Lustre: Skipped 2 previous similar messages [ 4693.474938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4693.478435] Lustre: Skipped 1 previous similar message [ 4702.175200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4702.203441] LustreError: 310043:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4702.208622] LustreError: 310043:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4702.327595] Lustre: server umount lustre-OST0000 complete [ 4707.296641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4707.300978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.307790] Lustre: Skipped 1 previous similar message [ 4707.311060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4707.314456] Lustre: Skipped 2 previous similar messages [ 4710.325117] Lustre: server umount lustre-MDT0000 complete [ 4712.048267] LustreError: 307885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817625 with bad export cookie 14465143331653847247 [ 4712.054739] LustreError: 307885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4712.055447] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4712.244139] Lustre: server umount lustre-MDT0001 complete [ 4716.183105] XFS (loop0): Unmounting Filesystem [ 4718.252453] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 02:13:50 (1759817630) [ 4721.472606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.724987] LustreError: 311266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4721.758075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.760747] Lustre: Skipped 2 previous similar messages [ 4723.314641] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4726.929691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4728.697933] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4731.254041] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4732.727019] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4735.608219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4738.269805] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4740.547139] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4757.477920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.478888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4757.484636] Lustre: Skipped 1 previous similar message [ 4757.494408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4757.497671] Lustre: Skipped 3 previous similar messages [ 4761.139141] LustreError: 313630:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4761.142732] LustreError: 313630:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4761.245846] Lustre: server umount lustre-OST0000 complete [ 4769.233701] Lustre: server umount lustre-MDT0000 complete [ 4770.803924] LustreError: 311245:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817684 with bad export cookie 14465143331653848129 [ 4770.808342] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4770.808715] LustreError: 311245:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4771.004812] Lustre: server umount lustre-MDT0001 complete [ 4775.698502] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4776.929429] Key type lgssc unregistered [ 4777.079566] LNet: 314566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4777.084274] LNetError: 314566:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4777.094648] LNet: Removed LNI 192.168.204.157@tcp [ 4777.606226] Key type .llcrypt unregistered [ 4777.609919] Key type ._llcrypt unregistered [ 4789.013331] Key type ._llcrypt registered [ 4789.014250] Key type .llcrypt registered [ 4789.057069] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4789.457661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4789.514106] alg: No test for adler32 (adler32-zlib) [ 4790.373652] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4790.459095] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4792.031233] Key type lgssc registered [ 4792.581264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4796.491201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4796.503888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4797.754225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.668424] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4803.666562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4803.821035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4805.760570] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4808.364609] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4809.970749] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4813.278254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4813.463533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.315564] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4818.820203] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4828.130928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4828.137964] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4828.140627] Lustre: Skipped 1 previous similar message [ 4829.176600] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.57@tcp (stopping) [ 4833.019873] LustreError: 318385:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4833.041395] Lustre: server umount lustre-MDT0001 complete [ 4833.249364] LustreError: 316091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4833.257710] LustreError: 316091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4834.006867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4834.279784] LustreError: 316085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4836.664166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4836.696517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4836.698091] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4836.711874] Lustre: 316077:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4836.848847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4838.224585] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4841.959566] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4846.052789] LustreError: 319565:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4846.055868] LustreError: 319565:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4846.059153] LustreError: 316082:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4846.072833] LustreError: 319568:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.157@tcp: cfg command failed: rc = -17 [ 4846.076393] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4846.076393] [ 5144.034582] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.035253] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5144.042672] Lustre: Skipped 1 previous similar message [ 5144.049286] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5144.053221] Lustre: Skipped 1 previous similar message [ 5146.605371] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.57@tcp (stopping) [ 5147.770826] LustreError: 330958:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5147.772683] LustreError: 330958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5147.794555] Lustre: server umount lustre-MDT0001 complete [ 5149.154304] LustreError: 316090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.162658] LustreError: 316090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5149.228093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5151.719040] LustreError: 317686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.318016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5152.352343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5152.354073] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5152.366348] Lustre: 316077:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5152.484718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5154.033200] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 5157.869120] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5161.966441] LustreError: 332135:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5161.971315] LustreError: 332135:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5161.976714] LustreError: 316082:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5161.980679] LustreError: 316082:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5162.010829] LustreError: 332138:0:(obd_config.c:689:class_setup()) Device 8 already setup (type osp) [ 5162.015247] LustreError: 332138:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.157@tcp: cfg command failed: rc = -17 [ 5162.020917] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.157@tcp [ 5162.020917] [ 5458.927373] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.57@tcp (stopping) [ 5459.939103] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.939865] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5459.947400] Lustre: Skipped 1 previous similar message [ 5462.801965] LustreError: 342923:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5462.806185] LustreError: 342923:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5462.837163] Lustre: server umount lustre-MDT0001 complete [ 5463.877511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5464.039615] LustreError: 317686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.461077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5466.494010] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5466.495912] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5466.508760] Lustre: 316079:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5466.623959] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5467.922354] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 5471.717481] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5472.230574] LustreError: 343999:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5472.234238] LustreError: 343999:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5472.238185] LustreError: 316082:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5472.241468] LustreError: 316082:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5472.259019] LustreError: 344002:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 5472.267412] LustreError: 344002:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.157@tcp: cfg command failed: rc = -114 [ 5472.270116] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 5472.270116] [ 5784.032621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5784.033684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.036090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5784.038366] Lustre: Skipped 3 previous similar messages [ 5784.040398] Lustre: Skipped 3 previous similar messages [ 5788.544991] LustreError: 355007:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5788.547324] LustreError: 355007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5788.584295] Lustre: server umount lustre-MDT0000 complete [ 5789.155059] LustreError: 316086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.159038] LustreError: 316086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5789.863695] LustreError: 316069:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818703 with bad export cookie 4195756902680164312 [ 5789.870692] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.273366] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5796.049370] LustreError: 355208:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5796.053226] LustreError: 355208:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5796.169370] Lustre: server umount lustre-MDT0001 complete [ 5797.775080] Lustre: server umount lustre-OST0000 complete [ 5801.081880] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 5804.135877] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 5808.294589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5810.703929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5813.184943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5816.202261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5818.499405] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 02:32:11 (1759818731) [ 5821.282815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5821.311947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5821.410123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5821.424642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5821.459605] Lustre: lustre-MDT0000: new disk, initializing [ 5821.485426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.492071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5823.004308] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 5828.330274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5828.355846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5828.388485] Lustre: 357939:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5828.414844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5828.416506] Lustre: Skipped 1 previous similar message [ 5828.450825] Lustre: lustre-MDT0001: new disk, initializing [ 5828.482923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5828.495464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5828.498629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5830.261834] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 5834.033420] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5835.378957] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5838.415786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5838.456042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5838.576583] LustreError: 359671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.580491] LustreError: 359671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5838.583994] Lustre: lustre-OST0000: new disk, initializing [ 5838.586200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5838.610612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.502984] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 5843.944778] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5843.957054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5843.960081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5843.968617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5851.105280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.108721] Lustre: Skipped 2 previous similar messages [ 5851.109686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.111500] Lustre: Skipped 4 previous similar messages [ 5856.971281] LustreError: 360384:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5856.976548] LustreError: 360384:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5857.019931] Lustre: server umount lustre-MDT0000 complete [ 5858.406787] LustreError: 357931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818771 with bad export cookie 4195756902680166286 [ 5858.410491] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5858.410893] LustreError: 357931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5858.411506] LustreError: 358730:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 5858.550879] Lustre: server umount lustre-MDT0001 complete [ 5870.332785] Lustre: server umount lustre-OST0000 complete [ 5880.749655] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 5884.143189] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 5888.747569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5891.922720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5894.589391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5897.378767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5897.958332] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5754 sec ========== 02:33:30 (1759818810) [ 5898.644550] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:33:31 (1759818811) === [ 5899.953300] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:33:32 (1759818812) === [ 5915.322533] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 5916.622086] Key type lgssc unregistered [ 5916.764476] LNet: 365889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5916.768288] LNetError: 365889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5916.779522] LNet: Removed LNI 192.168.204.157@tcp [ 5917.211768] Key type .llcrypt unregistered [ 5917.213783] Key type ._llcrypt unregistered